[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:53:09 dist-test-slave-tz51 krb5kdc[14615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:53:09 dist-test-slave-tz51 krb5kdc[14615](info): set up 2 sockets
Sep 27 10:53:09 dist-test-slave-tz51 krb5kdc[14615](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 10:53:10.523619 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:53:10 dist-test-slave-tz51 krb5kdc[14615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:53:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:53:10.648089 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:39081
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:39081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:10.760614 14628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:10.760918 14628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:10.760995 14628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:10.765067 14628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:10.765144 14628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:10.765180 14628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:10.765236 14628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:10.765275 14628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:10.770306 14628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:39081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:39081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:10.771288 14628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:10.772281 14628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:10.780494 14636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:10.780515 14634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:10.780514 14633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:10.780772 14628 server_base.cc:1034] running on GCE node
I20260927 10:53:10.781286 14628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:10.782370 14628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:10.783520 14628 hybrid_clock.cc:648] HybridClock initialized: now 1790506390783495 us; error 39 us; skew 500 ppm
Sep 27 10:53:10 dist-test-slave-tz51 krb5kdc[14615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506390, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:10.786201 14628 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:53:10.787094 14628 webserver.cc:460] Webserver started at http://127.14.68.62:32867/ using document root <none> and password file <none>
I20260927 10:53:10.787663 14628 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:10.787781 14628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:10.787999 14628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:10.789944 14628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "4e1b1aadec2b4dbfa6a65a2946c82af9"
format_stamp: "Formatted at 2026-09-27 10:53:10 on dist-test-slave-tz51"
server_key: "dcd94bbd94b7117b69a677f8777ea6be"
server_key_iv: "120d25e219df8a7447f1d26f57afd6d4"
server_key_version: "encryptionkey@0"
I20260927 10:53:10.790524 14628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "4e1b1aadec2b4dbfa6a65a2946c82af9"
format_stamp: "Formatted at 2026-09-27 10:53:10 on dist-test-slave-tz51"
server_key: "dcd94bbd94b7117b69a677f8777ea6be"
server_key_iv: "120d25e219df8a7447f1d26f57afd6d4"
server_key_version: "encryptionkey@0"
I20260927 10:53:10.794224 14628 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:53:10.796653 14643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:10.797823 14628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:10.797959 14628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "4e1b1aadec2b4dbfa6a65a2946c82af9"
format_stamp: "Formatted at 2026-09-27 10:53:10 on dist-test-slave-tz51"
server_key: "dcd94bbd94b7117b69a677f8777ea6be"
server_key_iv: "120d25e219df8a7447f1d26f57afd6d4"
server_key_version: "encryptionkey@0"
I20260927 10:53:10.798064 14628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:10.814400 14628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:10.817847 14628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:10.818079 14628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:10.825290 14628 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:39081
I20260927 10:53:10.825592 14695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:39081 every 8 connection(s)
I20260927 10:53:10.826313 14628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:10.829166 14696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:10.834264 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 14628
I20260927 10:53:10.834436 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:53:10.834539 14696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e1b1aadec2b4dbfa6a65a2946c82af9: Bootstrap starting.
I20260927 10:53:10.835791 14608 external_mini_cluster.cc:1428] Setting key f6f36197be9d3b51438c5dd25d548c94
I20260927 10:53:10.837062 14696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e1b1aadec2b4dbfa6a65a2946c82af9: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:10.837809 14696 log.cc:824] T 00000000000000000000000000000000 P 4e1b1aadec2b4dbfa6a65a2946c82af9: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:10.840142 14696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e1b1aadec2b4dbfa6a65a2946c82af9: No bootstrap required, opened a new log
I20260927 10:53:10.845228 14696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e1b1aadec2b4dbfa6a65a2946c82af9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1b1aadec2b4dbfa6a65a2946c82af9" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 39081 } }
I20260927 10:53:10.845520 14696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e1b1aadec2b4dbfa6a65a2946c82af9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:10.845587 14696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e1b1aadec2b4dbfa6a65a2946c82af9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e1b1aadec2b4dbfa6a65a2946c82af9, State: Initialized, Role: FOLLOWER
I20260927 10:53:10.846096 14696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e1b1aadec2b4dbfa6a65a2946c82af9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1b1aadec2b4dbfa6a65a2946c82af9" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 39081 } }
I20260927 10:53:10.846268 14696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e1b1aadec2b4dbfa6a65a2946c82af9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:10.846336 14696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e1b1aadec2b4dbfa6a65a2946c82af9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:10.846438 14696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e1b1aadec2b4dbfa6a65a2946c82af9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:10.847683 14696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e1b1aadec2b4dbfa6a65a2946c82af9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1b1aadec2b4dbfa6a65a2946c82af9" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 39081 } }
I20260927 10:53:10.848078 14696 leader_election.cc:302] T 00000000000000000000000000000000 P 4e1b1aadec2b4dbfa6a65a2946c82af9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e1b1aadec2b4dbfa6a65a2946c82af9; no voters:
I20260927 10:53:10.848440 14696 leader_election.cc:288] T 00000000000000000000000000000000 P 4e1b1aadec2b4dbfa6a65a2946c82af9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:10.848548 14701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e1b1aadec2b4dbfa6a65a2946c82af9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:10.848794 14701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e1b1aadec2b4dbfa6a65a2946c82af9 [term 1 LEADER]: Becoming Leader. State: Replica: 4e1b1aadec2b4dbfa6a65a2946c82af9, State: Running, Role: LEADER
I20260927 10:53:10.849107 14701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e1b1aadec2b4dbfa6a65a2946c82af9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1b1aadec2b4dbfa6a65a2946c82af9" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 39081 } }
I20260927 10:53:10.849725 14696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e1b1aadec2b4dbfa6a65a2946c82af9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:10.850762 14703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e1b1aadec2b4dbfa6a65a2946c82af9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e1b1aadec2b4dbfa6a65a2946c82af9. Latest consensus state: current_term: 1 leader_uuid: "4e1b1aadec2b4dbfa6a65a2946c82af9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1b1aadec2b4dbfa6a65a2946c82af9" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 39081 } } }
I20260927 10:53:10.850885 14703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e1b1aadec2b4dbfa6a65a2946c82af9 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:10.851248 14702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e1b1aadec2b4dbfa6a65a2946c82af9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e1b1aadec2b4dbfa6a65a2946c82af9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1b1aadec2b4dbfa6a65a2946c82af9" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 39081 } } }
I20260927 10:53:10.851358 14702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e1b1aadec2b4dbfa6a65a2946c82af9 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:10.851454 14710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:10.854630 14710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 10:53:10 dist-test-slave-tz51 krb5kdc[14615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:10.860743 14710 catalog_manager.cc:1347] Generated new cluster ID: 9884c3383ab3472a9703dba0bb276707
I20260927 10:53:10.860854 14710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:10.864122 14699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:10.838758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39006 (local address 127.14.68.62:39081)
0927 10:53:10.839203 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:10.839223 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:10.840663 (+ 1440us) server_negotiation.cc:407] Connection header received
0927 10:53:10.844825 (+ 4162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:10.844842 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:10.845170 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:10.845455 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:10.848659 (+ 3204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:10.850077 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:10.851999 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:10.852779 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:10.852933 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:10.858264 (+ 5331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:10.858270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:10.858287 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:10.858319 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:10.860292 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:10.861087 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:10.861093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:10.861098 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:10.861247 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:10.861681 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:10.861684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:10.861686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:10.862135 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:10.862331 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:10.862991 (+ 660us) server_negotiation.cc:299] Negotiation successful
0927 10:53:10.863257 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":114,"threads_started":1}
I20260927 10:53:10.867645 14710 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:10.868775 14710 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:10.876708 14710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e1b1aadec2b4dbfa6a65a2946c82af9: Generated new TSK 0
I20260927 10:53:10.877384 14710 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
I20260927 10:53:10.908846 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:39081
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:11.022461 14722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:11.022784 14722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:11.022866 14722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:11.026902 14722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:11.027001 14722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:11.027097 14722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:53:11.032368 14722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:39081
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:11.033457 14722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:11.034484 14722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:11.043910 14730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:11.043957 14727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:11.043982 14722 server_base.cc:1034] running on GCE node
W20260927 10:53:11.043926 14728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:11.044698 14722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:11.045339 14722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:11.046546 14722 hybrid_clock.cc:648] HybridClock initialized: now 1790506391046519 us; error 37 us; skew 500 ppm
Sep 27 10:53:11 dist-test-slave-tz51 krb5kdc[14615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506391, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:11.049428 14722 init.cc:377] Logged in from keytab as kudu/127.14.68.1@KRBTEST.COM (short username kudu)
I20260927 10:53:11.050427 14722 webserver.cc:460] Webserver started at http://127.14.68.1:40481/ using document root <none> and password file <none>
I20260927 10:53:11.050979 14722 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:11.051070 14722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:11.051376 14722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:11.053452 14722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "3b9d633afdc042baa576f923d624f01c"
format_stamp: "Formatted at 2026-09-27 10:53:11 on dist-test-slave-tz51"
server_key: "4a3e881b78967d2d1a0122f691aeb130"
server_key_iv: "33664ff5ddfaf8c85c362da21fa3b6c9"
server_key_version: "encryptionkey@0"
I20260927 10:53:11.054006 14722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "3b9d633afdc042baa576f923d624f01c"
format_stamp: "Formatted at 2026-09-27 10:53:11 on dist-test-slave-tz51"
server_key: "4a3e881b78967d2d1a0122f691aeb130"
server_key_iv: "33664ff5ddfaf8c85c362da21fa3b6c9"
server_key_version: "encryptionkey@0"
I20260927 10:53:11.057658 14722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:53:11.059948 14737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:11.061167 14722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:11.061328 14722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "3b9d633afdc042baa576f923d624f01c"
format_stamp: "Formatted at 2026-09-27 10:53:11 on dist-test-slave-tz51"
server_key: "4a3e881b78967d2d1a0122f691aeb130"
server_key_iv: "33664ff5ddfaf8c85c362da21fa3b6c9"
server_key_version: "encryptionkey@0"
I20260927 10:53:11.061441 14722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:11.071789 14722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:11.075027 14722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:11.075235 14722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:11.075897 14722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:11.076732 14722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:11.076800 14722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:11.076871 14722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:11.076922 14722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:11.085635 14722 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:46453
I20260927 10:53:11.085659 14850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:46453 every 8 connection(s)
I20260927 10:53:11.086494 14722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:53:11.095145 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 14722
I20260927 10:53:11.095238 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:53:11.095484 14608 external_mini_cluster.cc:1428] Setting key 6014a23152bc5707302b08dcbb849b1a
Sep 27 10:53:11 dist-test-slave-tz51 krb5kdc[14615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506391, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:11.102006 14699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:11.088375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:37533 (local address 127.14.68.62:39081)
0927 10:53:11.088520 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:11.088523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:11.089546 (+ 1023us) server_negotiation.cc:407] Connection header received
0927 10:53:11.090438 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:11.090440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:11.090486 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:11.090560 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:11.091773 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:11.092318 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:11.093454 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:11.093702 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:11.093848 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:11.096527 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:11.096531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:11.096533 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:11.096555 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:11.099462 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:11.100142 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:11.100145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:11.100146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:11.100221 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:11.100747 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:11.100754 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:11.100759 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:11.101024 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:11.101150 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:11.101687 (+ 537us) server_negotiation.cc:299] Negotiation successful
0927 10:53:11.101829 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:53:11.102957 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:11.088756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:39081 (local address 127.14.68.1:37533)
0927 10:53:11.089383 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:11.089424 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:11.090232 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:11.090724 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:11.090731 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:11.091084 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:11.091576 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:11.091589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:11.092442 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:11.092445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:11.093338 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:11.093348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:11.093892 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:11.093896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:11.094065 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:11.094687 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:11.094702 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:11.096370 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:11.099602 (+ 3232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:11.099612 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:11.099631 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:11.100037 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:11.100313 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:11.100319 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:11.100324 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:11.100516 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:11.101189 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:11.101195 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:11.101437 (+ 242us) client_negotiation.cc:769] Sending connection context
0927 10:53:11.101646 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 10:53:11.101970 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":110,"threads_started":1}
I20260927 10:53:11.104256 14851 heartbeater.cc:347] Connected to a master server at 127.14.68.62:39081
I20260927 10:53:11.104534 14851 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.14.68.2@KRBTEST.COM; defaulting to no policy
I20260927 10:53:11.105147 14851 heartbeater.cc:511] Master 127.14.68.62:39081 requested a full tablet report, sending...
I20260927 10:53:11.106765 14654 ts_manager.cc:194] Registered new tserver with Master: 3b9d633afdc042baa576f923d624f01c (127.14.68.1:46453)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.2@KRBTEST.COM" created.
I20260927 10:53:11.108116 14654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:37533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
I20260927 10:53:11.123523 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:39081
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:11.232641 14856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:11.232903 14856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:11.232949 14856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:11.236927 14856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:11.236979 14856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:11.237074 14856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:53:11.242281 14856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:39081
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:11.243283 14856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:11.244306 14856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:11.253929 14864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:11.253871 14862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:11.254176 14856 server_base.cc:1034] running on GCE node
W20260927 10:53:11.253871 14861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:11.254676 14856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:11.255280 14856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:11.256430 14856 hybrid_clock.cc:648] HybridClock initialized: now 1790506391256418 us; error 24 us; skew 500 ppm
Sep 27 10:53:11 dist-test-slave-tz51 krb5kdc[14615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506391, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:11.259029 14856 init.cc:377] Logged in from keytab as kudu/127.14.68.2@KRBTEST.COM (short username kudu)
I20260927 10:53:11.259946 14856 webserver.cc:460] Webserver started at http://127.14.68.2:36489/ using document root <none> and password file <none>
I20260927 10:53:11.260488 14856 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:11.260545 14856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:11.260774 14856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:11.262778 14856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "63ce94e944fa48c5a6ad199cf0a6fd2a"
format_stamp: "Formatted at 2026-09-27 10:53:11 on dist-test-slave-tz51"
server_key: "78a1525e7157a9de681fcf19b48c476c"
server_key_iv: "d5025172b5e2d981e3568604092f0685"
server_key_version: "encryptionkey@0"
I20260927 10:53:11.263337 14856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "63ce94e944fa48c5a6ad199cf0a6fd2a"
format_stamp: "Formatted at 2026-09-27 10:53:11 on dist-test-slave-tz51"
server_key: "78a1525e7157a9de681fcf19b48c476c"
server_key_iv: "d5025172b5e2d981e3568604092f0685"
server_key_version: "encryptionkey@0"
I20260927 10:53:11.267040 14856 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:53:11.269444 14871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:11.270643 14856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 10:53:11.270764 14856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "63ce94e944fa48c5a6ad199cf0a6fd2a"
format_stamp: "Formatted at 2026-09-27 10:53:11 on dist-test-slave-tz51"
server_key: "78a1525e7157a9de681fcf19b48c476c"
server_key_iv: "d5025172b5e2d981e3568604092f0685"
server_key_version: "encryptionkey@0"
I20260927 10:53:11.270864 14856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:11.284291 14856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:11.287072 14856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:11.287272 14856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:11.287856 14856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:11.288635 14856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:11.288679 14856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:11.288741 14856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:11.288774 14856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:11.297770 14856 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:46261
I20260927 10:53:11.297796 14984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:46261 every 8 connection(s)
I20260927 10:53:11.298575 14856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:53:11.299525 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 14856
I20260927 10:53:11.299609 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:53:11.299901 14608 external_mini_cluster.cc:1428] Setting key 528b78745b7d83f44235e5339ea66d46
Sep 27 10:53:11 dist-test-slave-tz51 krb5kdc[14615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506391, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.3@KRBTEST.COM" created.
I20260927 10:53:11.312852 14699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:11.300378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:34029 (local address 127.14.68.62:39081)
0927 10:53:11.300531 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:11.300535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:11.301669 (+ 1134us) server_negotiation.cc:407] Connection header received
0927 10:53:11.302625 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:11.302628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:11.302686 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:11.302761 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:11.303991 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:11.304872 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:11.306140 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:11.306573 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:11.306704 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:11.309044 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:11.309059 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:11.309062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:11.309094 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:11.310647 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:11.311172 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:11.311177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:11.311179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:11.311287 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:11.311665 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:11.311669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:11.311671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:11.311956 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:11.312062 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:11.312638 (+ 576us) server_negotiation.cc:299] Negotiation successful
0927 10:53:11.312737 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:53:11.313952 14987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:11.300830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:39081 (local address 127.14.68.2:34029)
0927 10:53:11.301511 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:11.301568 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:11.302429 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:11.302952 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:11.302958 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:11.303301 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:11.303837 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:11.303859 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:11.305029 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:11.305032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:11.305997 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:11.306007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:11.306718 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:11.306720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:11.306854 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:11.307436 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:11.307451 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:11.308871 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:11.310792 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:11.310798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:11.310811 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:11.311063 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:11.311434 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:11.311437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:11.311439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:11.311567 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:11.312062 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:11.312069 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:11.312348 (+ 279us) client_negotiation.cc:769] Sending connection context
0927 10:53:11.312574 (+ 226us) client_negotiation.cc:240] Negotiation successful
0927 10:53:11.312925 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":137,"threads_started":1}
I20260927 10:53:11.315071 14985 heartbeater.cc:347] Connected to a master server at 127.14.68.62:39081
I20260927 10:53:11.315310 14985 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:11.315886 14985 heartbeater.cc:511] Master 127.14.68.62:39081 requested a full tablet report, sending...
I20260927 10:53:11.316972 14654 ts_manager.cc:194] Registered new tserver with Master: 63ce94e944fa48c5a6ad199cf0a6fd2a (127.14.68.2:46261)
I20260927 10:53:11.317528 14654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:34029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
I20260927 10:53:11.328842 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:39081
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:11.441002 14990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:11.441258 14990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:11.441303 14990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:11.445381 14990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:11.445441 14990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:11.445508 14990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:53:11.451099 14990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:39081
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:11.452265 14990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:11.453328 14990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:11.463124 14995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:11.463352 14990 server_base.cc:1034] running on GCE node
W20260927 10:53:11.463127 14998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:11.463146 14996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:11.463944 14990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:11.464630 14990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:11.465842 14990 hybrid_clock.cc:648] HybridClock initialized: now 1790506391465819 us; error 39 us; skew 500 ppm
Sep 27 10:53:11 dist-test-slave-tz51 krb5kdc[14615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506391, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:11.468703 14990 init.cc:377] Logged in from keytab as kudu/127.14.68.3@KRBTEST.COM (short username kudu)
I20260927 10:53:11.469612 14990 webserver.cc:460] Webserver started at http://127.14.68.3:39299/ using document root <none> and password file <none>
I20260927 10:53:11.470188 14990 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:11.470280 14990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:11.470516 14990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:11.472688 14990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "85242093180c46a5ae9cc485983eac37"
format_stamp: "Formatted at 2026-09-27 10:53:11 on dist-test-slave-tz51"
server_key: "5287d11d64f30512318967c8b79ea177"
server_key_iv: "6a14cfd8bcf8701891a765f7d772d496"
server_key_version: "encryptionkey@0"
I20260927 10:53:11.473263 14990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "85242093180c46a5ae9cc485983eac37"
format_stamp: "Formatted at 2026-09-27 10:53:11 on dist-test-slave-tz51"
server_key: "5287d11d64f30512318967c8b79ea177"
server_key_iv: "6a14cfd8bcf8701891a765f7d772d496"
server_key_version: "encryptionkey@0"
I20260927 10:53:11.477195 14990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:53:11.479774 15005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:11.481030 14990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:11.481134 14990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "85242093180c46a5ae9cc485983eac37"
format_stamp: "Formatted at 2026-09-27 10:53:11 on dist-test-slave-tz51"
server_key: "5287d11d64f30512318967c8b79ea177"
server_key_iv: "6a14cfd8bcf8701891a765f7d772d496"
server_key_version: "encryptionkey@0"
I20260927 10:53:11.481221 14990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:11.493389 14990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:11.496443 14990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:11.496594 14990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:11.497145 14990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:11.497915 14990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:11.497957 14990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:11.497997 14990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:11.498018 14990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:11.507332 14990 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:44505
I20260927 10:53:11.507354 15118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:44505 every 8 connection(s)
I20260927 10:53:11.508287 14990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:53:11.515164 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 14990
I20260927 10:53:11.515302 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:53:11.515544 14608 external_mini_cluster.cc:1428] Setting key 78adfb374ed92f381ba34de29db48b5d
Sep 27 10:53:11 dist-test-slave-tz51 krb5kdc[14615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506391, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:11.521747 14699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:11.510160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:44937 (local address 127.14.68.62:39081)
0927 10:53:11.510299 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:11.510303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:11.511326 (+ 1023us) server_negotiation.cc:407] Connection header received
0927 10:53:11.512252 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:11.512254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:11.512300 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:11.512374 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:11.513523 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:11.514069 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:11.515251 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:11.515572 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:11.515718 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:11.518631 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:11.518634 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:11.518636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:11.518659 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:11.519936 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:11.520417 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:11.520419 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:11.520420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:11.520494 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:11.520835 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:11.520838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:11.520839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:11.521027 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:11.521115 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:11.521535 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 10:53:11.521637 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:53:11.522593 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:11.510578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:39081 (local address 127.14.68.3:44937)
0927 10:53:11.511201 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:11.511242 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:11.512064 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:11.512536 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:11.512554 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:11.512906 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:11.513376 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:11.513389 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:11.514193 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:11.514196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:11.515122 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:11.515133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:11.515797 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:11.515802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:11.515959 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:11.516713 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:11.516734 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:11.518480 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:11.520043 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:11.520047 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:11.520058 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:11.520323 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:11.520587 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:11.520589 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:11.520590 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:11.520727 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:11.521124 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:11.521128 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:11.521344 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 10:53:11.521488 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 10:53:11.521737 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":109,"threads_started":1}
I20260927 10:53:11.523664 15119 heartbeater.cc:347] Connected to a master server at 127.14.68.62:39081
I20260927 10:53:11.523952 15119 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:11.524435 15119 heartbeater.cc:511] Master 127.14.68.62:39081 requested a full tablet report, sending...
I20260927 10:53:11.525415 14654 ts_manager.cc:194] Registered new tserver with Master: 85242093180c46a5ae9cc485983eac37 (127.14.68.3:44505)
I20260927 10:53:11.525965 14654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:44937
I20260927 10:53:11.529678 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 10:53:11 dist-test-slave-tz51 krb5kdc[14615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506391, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:53:11 dist-test-slave-tz51 krb5kdc[14615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506391, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.68.1@KRBTEST.COM
I20260927 10:53:11.555589 15126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:11.545668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.14.68.1:46453)
0927 10:53:11.546018 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:11.546027 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:11.546065 (+ 38us) server_negotiation.cc:407] Connection header received
0927 10:53:11.546193 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:11.546199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:11.546385 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:11.546504 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:11.547139 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:11.547971 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:11.548795 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:11.549191 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:11.549372 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:11.551968 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:11.551976 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:11.551989 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:11.552012 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:11.553765 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:11.554181 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:11.554187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:11.554188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:11.554264 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:11.554523 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:11.554526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:11.554527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:11.554824 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:11.554992 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:11.555212 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 10:53:11.555381 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":151,"threads_started":1}
W20260927 10:53:11.556654 14765 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51736
Sep 27 10:53:11 dist-test-slave-tz51 krb5kdc[14615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:53:11 dist-test-slave-tz51 krb5kdc[14615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.1@KRBTEST.COM
I20260927 10:53:11.587020 15126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:11.578273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51748 (local address 127.14.68.1:46453)
0927 10:53:11.578434 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:11.578439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:11.578561 (+ 122us) server_negotiation.cc:407] Connection header received
0927 10:53:11.578742 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:11.578746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:11.578800 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:11.578884 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:11.579593 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:11.580208 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:11.581067 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:11.581319 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:11.581434 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:11.583984 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:11.583988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:11.583991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:11.584017 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:11.585365 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:11.585779 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:11.585783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:11.585785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:11.585869 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:11.586168 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:11.586173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:11.586175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:11.586345 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:11.586479 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:11.586725 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 10:53:11.586853 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:53:11.588155 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 14722
I20260927 10:53:11.594460 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 14856
I20260927 10:53:11.600088 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 14990
I20260927 10:53:11.605265 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 14628
2026-09-27T10:53:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2139 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:53:11 dist-test-slave-tz51 krb5kdc[15133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:53:11 dist-test-slave-tz51 krb5kdc[15133](info): set up 2 sockets
Sep 27 10:53:11 dist-test-slave-tz51 krb5kdc[15133](info): commencing operation
krb5kdc: starting...
W20260927 10:53:12.643324 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:53:12 dist-test-slave-tz51 krb5kdc[15133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:53:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:53:12.766144 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:46619
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:33159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:46619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:12.878899 15146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:12.879151 15146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:12.879190 15146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:12.883126 15146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:12.883183 15146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:12.883201 15146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:12.883217 15146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:12.883232 15146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:12.888464 15146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:33159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:46619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:46619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:12.889425 15146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:12.890388 15146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:12.898689 15152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:12.898715 15151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:12.898689 15154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:12.898883 15146 server_base.cc:1034] running on GCE node
I20260927 10:53:12.899492 15146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:12.900660 15146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:12.901821 15146 hybrid_clock.cc:648] HybridClock initialized: now 1790506392901805 us; error 25 us; skew 500 ppm
Sep 27 10:53:12 dist-test-slave-tz51 krb5kdc[15133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506392, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:12.904627 15146 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:53:12.905450 15146 webserver.cc:460] Webserver started at http://127.14.68.62:43857/ using document root <none> and password file <none>
I20260927 10:53:12.906000 15146 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:12.906095 15146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:12.906304 15146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:12.908363 15146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "71d095b4f8864c9f89b5d807051dd156"
format_stamp: "Formatted at 2026-09-27 10:53:12 on dist-test-slave-tz51"
server_key: "317680ce66d561f2d923f2ac2817689f"
server_key_iv: "50795bee9ee5dfca3417cb784c7a889e"
server_key_version: "encryptionkey@0"
I20260927 10:53:12.908911 15146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "71d095b4f8864c9f89b5d807051dd156"
format_stamp: "Formatted at 2026-09-27 10:53:12 on dist-test-slave-tz51"
server_key: "317680ce66d561f2d923f2ac2817689f"
server_key_iv: "50795bee9ee5dfca3417cb784c7a889e"
server_key_version: "encryptionkey@0"
I20260927 10:53:12.912951 15146 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:53:12.915383 15161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:12.916649 15146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:12.916795 15146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "71d095b4f8864c9f89b5d807051dd156"
format_stamp: "Formatted at 2026-09-27 10:53:12 on dist-test-slave-tz51"
server_key: "317680ce66d561f2d923f2ac2817689f"
server_key_iv: "50795bee9ee5dfca3417cb784c7a889e"
server_key_version: "encryptionkey@0"
I20260927 10:53:12.916910 15146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:12.938243 15146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:12.941416 15146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:12.941723 15146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:12.948729 15146 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:46619
I20260927 10:53:12.948745 15213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:46619 every 8 connection(s)
I20260927 10:53:12.949718 15146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:12.952101 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 15146
I20260927 10:53:12.952234 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:53:12.952507 14608 external_mini_cluster.cc:1428] Setting key 1b5caae44cff4bd8f309d886023d42b5
I20260927 10:53:12.952939 15214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:12.960000 15214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71d095b4f8864c9f89b5d807051dd156: Bootstrap starting.
Sep 27 10:53:12 dist-test-slave-tz51 krb5kdc[15133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:12.962400 15214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71d095b4f8864c9f89b5d807051dd156: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:12.963317 15214 log.cc:824] T 00000000000000000000000000000000 P 71d095b4f8864c9f89b5d807051dd156: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:12.965483 15214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71d095b4f8864c9f89b5d807051dd156: No bootstrap required, opened a new log
I20260927 10:53:12.967620 15217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:12.953893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36806 (local address 127.14.68.62:46619)
0927 10:53:12.954315 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:12.954334 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:12.954376 (+ 42us) server_negotiation.cc:407] Connection header received
0927 10:53:12.955202 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:12.955232 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:12.955648 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:12.956052 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:12.956702 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:12.958178 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:12.959029 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:12.959722 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:12.959874 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:12.962544 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:12.962550 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:12.962565 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:12.962601 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:12.964582 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:12.964991 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:12.964997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:12.965002 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:12.965128 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:12.965389 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:12.965392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:12.965394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:12.965880 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:12.966074 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:12.966392 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 10:53:12.966670 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":113,"threads_started":1}
I20260927 10:53:12.969785 15214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71d095b4f8864c9f89b5d807051dd156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d095b4f8864c9f89b5d807051dd156" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46619 } }
I20260927 10:53:12.970069 15214 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71d095b4f8864c9f89b5d807051dd156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:12.970131 15214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71d095b4f8864c9f89b5d807051dd156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d095b4f8864c9f89b5d807051dd156, State: Initialized, Role: FOLLOWER
I20260927 10:53:12.970573 15214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71d095b4f8864c9f89b5d807051dd156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d095b4f8864c9f89b5d807051dd156" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46619 } }
I20260927 10:53:12.970729 15214 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71d095b4f8864c9f89b5d807051dd156 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:12.970800 15214 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71d095b4f8864c9f89b5d807051dd156 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:12.970896 15214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71d095b4f8864c9f89b5d807051dd156 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:12.972024 15214 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71d095b4f8864c9f89b5d807051dd156 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d095b4f8864c9f89b5d807051dd156" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46619 } }
I20260927 10:53:12.972365 15214 leader_election.cc:302] T 00000000000000000000000000000000 P 71d095b4f8864c9f89b5d807051dd156 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71d095b4f8864c9f89b5d807051dd156; no voters:
I20260927 10:53:12.972664 15214 leader_election.cc:288] T 00000000000000000000000000000000 P 71d095b4f8864c9f89b5d807051dd156 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:12.972795 15219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71d095b4f8864c9f89b5d807051dd156 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:12.973038 15219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71d095b4f8864c9f89b5d807051dd156 [term 1 LEADER]: Becoming Leader. State: Replica: 71d095b4f8864c9f89b5d807051dd156, State: Running, Role: LEADER
I20260927 10:53:12.973387 15219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71d095b4f8864c9f89b5d807051dd156 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d095b4f8864c9f89b5d807051dd156" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46619 } }
I20260927 10:53:12.973923 15214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71d095b4f8864c9f89b5d807051dd156 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:12.975013 15221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71d095b4f8864c9f89b5d807051dd156 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71d095b4f8864c9f89b5d807051dd156. Latest consensus state: current_term: 1 leader_uuid: "71d095b4f8864c9f89b5d807051dd156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d095b4f8864c9f89b5d807051dd156" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46619 } } }
I20260927 10:53:12.975136 15221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71d095b4f8864c9f89b5d807051dd156 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:12.975351 15220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71d095b4f8864c9f89b5d807051dd156 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71d095b4f8864c9f89b5d807051dd156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d095b4f8864c9f89b5d807051dd156" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46619 } } }
I20260927 10:53:12.975492 15220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71d095b4f8864c9f89b5d807051dd156 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:12.976025 15228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:12.979480 15228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:12.985737 15228 catalog_manager.cc:1347] Generated new cluster ID: 9a4d36d8e6b843a39f2c308d3c096fcd
I20260927 10:53:12.985837 15228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:13.007045 15228 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:13.008062 15228 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:13.017565 15228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71d095b4f8864c9f89b5d807051dd156: Generated new TSK 0
I20260927 10:53:13.018436 15228 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
I20260927 10:53:13.054821 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46619
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:33159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:13.162639 15240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:13.162879 15240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:13.162921 15240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:13.166769 15240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:13.166817 15240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:13.166873 15240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:53:13.171888 15240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:33159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46619
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:13.172825 15240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:13.173733 15240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:53:13.183422 15240 server_base.cc:1034] running on GCE node
W20260927 10:53:13.183343 15246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:13.183362 15248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:13.183339 15245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:13.184060 15240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:13.184652 15240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:13.185801 15240 hybrid_clock.cc:648] HybridClock initialized: now 1790506393185770 us; error 45 us; skew 500 ppm
Sep 27 10:53:13 dist-test-slave-tz51 krb5kdc[15133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506393, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:13.188616 15240 init.cc:377] Logged in from keytab as kudu/127.14.68.1@KRBTEST.COM (short username kudu)
I20260927 10:53:13.189368 15240 webserver.cc:460] Webserver started at http://127.14.68.1:46231/ using document root <none> and password file <none>
I20260927 10:53:13.189883 15240 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:13.189940 15240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:13.190093 15240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:13.192034 15240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "9650258f3b214454ae28cc4c1ed0fced"
format_stamp: "Formatted at 2026-09-27 10:53:13 on dist-test-slave-tz51"
server_key: "064bb9397facacd8a7da20e10d3ba8e3"
server_key_iv: "e5c2c3d0317af75f52dfdd01ed95db20"
server_key_version: "encryptionkey@0"
I20260927 10:53:13.192544 15240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "9650258f3b214454ae28cc4c1ed0fced"
format_stamp: "Formatted at 2026-09-27 10:53:13 on dist-test-slave-tz51"
server_key: "064bb9397facacd8a7da20e10d3ba8e3"
server_key_iv: "e5c2c3d0317af75f52dfdd01ed95db20"
server_key_version: "encryptionkey@0"
I20260927 10:53:13.196125 15240 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:53:13.198382 15255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:13.199558 15240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:13.199731 15240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "9650258f3b214454ae28cc4c1ed0fced"
format_stamp: "Formatted at 2026-09-27 10:53:13 on dist-test-slave-tz51"
server_key: "064bb9397facacd8a7da20e10d3ba8e3"
server_key_iv: "e5c2c3d0317af75f52dfdd01ed95db20"
server_key_version: "encryptionkey@0"
I20260927 10:53:13.199851 15240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:13.225317 15240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:13.228633 15240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:13.228819 15240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:13.229404 15240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:13.230228 15240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:13.230293 15240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:13.230360 15240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:13.230404 15240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:13.239240 15240 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:33213
I20260927 10:53:13.239267 15368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:33213 every 8 connection(s)
I20260927 10:53:13.240142 15240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:53:13.240751 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 15240
I20260927 10:53:13.240868 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:53:13.241137 14608 external_mini_cluster.cc:1428] Setting key 2c619313558686f28df00acb271182c9
Sep 27 10:53:13 dist-test-slave-tz51 krb5kdc[15133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506393, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.2@KRBTEST.COM" created.
I20260927 10:53:13.254729 15217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:13.241957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:43095 (local address 127.14.68.62:46619)
0927 10:53:13.242122 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:13.242127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:13.243025 (+ 898us) server_negotiation.cc:407] Connection header received
0927 10:53:13.244600 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:13.244602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:13.244660 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:13.244729 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:13.245999 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.246585 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.247831 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.248078 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.248193 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:13.250873 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:13.250876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:13.250878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:13.250901 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:13.252563 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:13.253151 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:13.253154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:13.253156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:13.253228 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:13.253578 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:13.253581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:13.253583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:13.253816 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:13.253924 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:13.254426 (+ 502us) server_negotiation.cc:299] Negotiation successful
0927 10:53:13.254551 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 10:53:13.255549 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:13.242364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:46619 (local address 127.14.68.1:43095)
0927 10:53:13.242857 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:13.242907 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:13.244368 (+ 1461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:13.244914 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:13.244927 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:13.245368 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:13.245836 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:13.245850 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.246742 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.246747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:13.247629 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:13.247639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.248233 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.248237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:13.248409 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:13.249036 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:13.249052 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:13.250700 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:13.252693 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:13.252698 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:13.252713 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:13.253021 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:13.253320 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:13.253324 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:13.253326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:13.253460 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:13.253925 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:13.253933 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:13.254176 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 10:53:13.254369 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 10:53:13.254692 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":135,"threads_started":1}
I20260927 10:53:13.256922 15369 heartbeater.cc:347] Connected to a master server at 127.14.68.62:46619
I20260927 10:53:13.257165 15369 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:13.257647 15369 heartbeater.cc:511] Master 127.14.68.62:46619 requested a full tablet report, sending...
I20260927 10:53:13.259328 15178 ts_manager.cc:194] Registered new tserver with Master: 9650258f3b214454ae28cc4c1ed0fced (127.14.68.1:33213)
I20260927 10:53:13.261008 15178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:43095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
I20260927 10:53:13.270849 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46619
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:33159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:13.379733 15374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:13.379971 15374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:13.380015 15374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:13.383828 15374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:13.383874 15374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:13.383930 15374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:53:13.389067 15374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:33159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46619
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:13.389957 15374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:13.390874 15374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:53:13.401028 15374 server_base.cc:1034] running on GCE node
W20260927 10:53:13.400960 15382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:13.400960 15380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:13.400960 15379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:13.401677 15374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:13.402379 15374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:13.403574 15374 hybrid_clock.cc:648] HybridClock initialized: now 1790506393403538 us; error 47 us; skew 500 ppm
Sep 27 10:53:13 dist-test-slave-tz51 krb5kdc[15133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506393, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:13.406688 15374 init.cc:377] Logged in from keytab as kudu/127.14.68.2@KRBTEST.COM (short username kudu)
I20260927 10:53:13.407513 15374 webserver.cc:460] Webserver started at http://127.14.68.2:37837/ using document root <none> and password file <none>
I20260927 10:53:13.408118 15374 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:13.408212 15374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:13.408420 15374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:13.410418 15374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "36ea75ab11d14c2e877d49c5a33645bf"
format_stamp: "Formatted at 2026-09-27 10:53:13 on dist-test-slave-tz51"
server_key: "dcb4610066eb6f87495633740e2cd3bb"
server_key_iv: "7b6550f20ce54a47b1a7e89f125da9b8"
server_key_version: "encryptionkey@0"
I20260927 10:53:13.410961 15374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "36ea75ab11d14c2e877d49c5a33645bf"
format_stamp: "Formatted at 2026-09-27 10:53:13 on dist-test-slave-tz51"
server_key: "dcb4610066eb6f87495633740e2cd3bb"
server_key_iv: "7b6550f20ce54a47b1a7e89f125da9b8"
server_key_version: "encryptionkey@0"
I20260927 10:53:13.414639 15374 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:53:13.417176 15389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:13.418360 15374 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:53:13.418535 15374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "36ea75ab11d14c2e877d49c5a33645bf"
format_stamp: "Formatted at 2026-09-27 10:53:13 on dist-test-slave-tz51"
server_key: "dcb4610066eb6f87495633740e2cd3bb"
server_key_iv: "7b6550f20ce54a47b1a7e89f125da9b8"
server_key_version: "encryptionkey@0"
I20260927 10:53:13.418681 15374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:13.431617 15374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:13.434465 15374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:13.434669 15374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:13.435326 15374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:13.436251 15374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:13.436403 15374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:13.436517 15374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:13.436592 15374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:13.445446 15374 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:45625
I20260927 10:53:13.445469 15502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:45625 every 8 connection(s)
I20260927 10:53:13.446328 15374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:53:13.447085 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 15374
I20260927 10:53:13.447209 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:53:13.447487 14608 external_mini_cluster.cc:1428] Setting key f69e4b2a4cc145ad637c195e2406f991
Sep 27 10:53:13 dist-test-slave-tz51 krb5kdc[15133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506393, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.3@KRBTEST.COM" created.
I20260927 10:53:13.461488 15217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:13.448231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:51013 (local address 127.14.68.62:46619)
0927 10:53:13.448364 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:13.448367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:13.449803 (+ 1436us) server_negotiation.cc:407] Connection header received
0927 10:53:13.451140 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:13.451143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:13.451197 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:13.451273 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:13.452506 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.453381 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.454532 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.454978 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.455110 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:13.457650 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:13.457656 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:13.457661 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:13.457696 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:13.459209 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:13.459751 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:13.459755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:13.459757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:13.459862 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:13.460193 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:13.460195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:13.460196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:13.460431 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:13.460547 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:13.461073 (+ 526us) server_negotiation.cc:299] Negotiation successful
0927 10:53:13.461225 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:53:13.462143 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:13.448764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:46619 (local address 127.14.68.2:51013)
0927 10:53:13.449634 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:13.449693 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:13.450893 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:13.451443 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:13.451450 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:13.451884 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:13.452354 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:13.452366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.453519 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.453522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:13.454406 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:13.454415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.455166 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.455169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:13.455314 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:13.455992 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:13.456008 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:13.457465 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:13.459309 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:13.459334 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:13.459346 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:13.459608 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:13.459955 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:13.459957 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:13.459958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:13.460105 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:13.460539 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:13.460544 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:13.460817 (+ 273us) client_negotiation.cc:769] Sending connection context
0927 10:53:13.460991 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 10:53:13.461252 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":143,"threads_started":1}
I20260927 10:53:13.463498 15503 heartbeater.cc:347] Connected to a master server at 127.14.68.62:46619
I20260927 10:53:13.463809 15503 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:13.464323 15503 heartbeater.cc:511] Master 127.14.68.62:46619 requested a full tablet report, sending...
I20260927 10:53:13.465425 15178 ts_manager.cc:194] Registered new tserver with Master: 36ea75ab11d14c2e877d49c5a33645bf (127.14.68.2:45625)
I20260927 10:53:13.466001 15178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:51013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
I20260927 10:53:13.476825 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46619
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:33159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:13.586835 15508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:13.587142 15508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:13.587222 15508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:13.591329 15508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:13.591423 15508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:13.591517 15508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:53:13.596787 15508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:33159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46619
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:13.597801 15508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:13.598789 15508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:13.608217 15513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:13.608316 15516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:13.608377 15514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:13.608433 15508 server_base.cc:1034] running on GCE node
I20260927 10:53:13.608863 15508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:13.609416 15508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:13.610591 15508 hybrid_clock.cc:648] HybridClock initialized: now 1790506393610562 us; error 40 us; skew 500 ppm
Sep 27 10:53:13 dist-test-slave-tz51 krb5kdc[15133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506393, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:13.613523 15508 init.cc:377] Logged in from keytab as kudu/127.14.68.3@KRBTEST.COM (short username kudu)
I20260927 10:53:13.614375 15508 webserver.cc:460] Webserver started at http://127.14.68.3:34459/ using document root <none> and password file <none>
I20260927 10:53:13.614938 15508 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:13.615000 15508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:13.615214 15508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:13.617254 15508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "45ad276ff8304f45b5122bd1887a1a39"
format_stamp: "Formatted at 2026-09-27 10:53:13 on dist-test-slave-tz51"
server_key: "be3a881320a0a9074bf53af13230ac73"
server_key_iv: "063cf643792fc5dc0dfe746021d338a3"
server_key_version: "encryptionkey@0"
I20260927 10:53:13.617801 15508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "45ad276ff8304f45b5122bd1887a1a39"
format_stamp: "Formatted at 2026-09-27 10:53:13 on dist-test-slave-tz51"
server_key: "be3a881320a0a9074bf53af13230ac73"
server_key_iv: "063cf643792fc5dc0dfe746021d338a3"
server_key_version: "encryptionkey@0"
I20260927 10:53:13.621658 15508 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:53:13.624264 15523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:13.625576 15508 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:13.625718 15508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "45ad276ff8304f45b5122bd1887a1a39"
format_stamp: "Formatted at 2026-09-27 10:53:13 on dist-test-slave-tz51"
server_key: "be3a881320a0a9074bf53af13230ac73"
server_key_iv: "063cf643792fc5dc0dfe746021d338a3"
server_key_version: "encryptionkey@0"
I20260927 10:53:13.625842 15508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:13.646382 15508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:13.649693 15508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:13.649920 15508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:13.650571 15508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:13.651405 15508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:13.651477 15508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:13.651546 15508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:13.651595 15508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:13.660650 15508 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:36605
I20260927 10:53:13.660676 15636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:36605 every 8 connection(s)
I20260927 10:53:13.661531 15508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:53:13.662931 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 15508
I20260927 10:53:13.663064 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:53:13.663383 14608 external_mini_cluster.cc:1428] Setting key 9410a2390a8a832d61df10db181a8659
Sep 27 10:53:13 dist-test-slave-tz51 krb5kdc[15133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506393, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:13.674510 15217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:13.663400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:57007 (local address 127.14.68.62:46619)
0927 10:53:13.663552 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:13.663556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:13.664575 (+ 1019us) server_negotiation.cc:407] Connection header received
0927 10:53:13.665626 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:13.665629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:13.665687 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:13.665752 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:13.666872 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.667393 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.668556 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.668771 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.668850 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:13.671157 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:13.671160 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:13.671162 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:13.671185 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:13.672610 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:13.673103 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:13.673106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:13.673107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:13.673179 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:13.673512 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:13.673515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:13.673518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:13.673704 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:13.673776 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:13.674210 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 10:53:13.674351 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 10:53:13.675300 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:13.663875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:46619 (local address 127.14.68.3:57007)
0927 10:53:13.664377 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:13.664442 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:13.665433 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:13.665920 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:13.665926 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:13.666281 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:13.666735 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:13.666748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.667504 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.667506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:13.668444 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:13.668453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.668891 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.668894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:13.669028 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:13.669552 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:13.669567 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:13.670970 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:13.672720 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:13.672725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:13.672736 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:13.672987 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:13.673266 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:13.673269 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:13.673270 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:13.673397 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:13.673789 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:13.673794 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:13.674002 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 10:53:13.674164 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 10:53:13.674480 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":153,"threads_started":1}
I20260927 10:53:13.676489 15637 heartbeater.cc:347] Connected to a master server at 127.14.68.62:46619
I20260927 10:53:13.676729 15637 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:13.677299 15637 heartbeater.cc:511] Master 127.14.68.62:46619 requested a full tablet report, sending...
I20260927 10:53:13.678354 15178 ts_manager.cc:194] Registered new tserver with Master: 45ad276ff8304f45b5122bd1887a1a39 (127.14.68.3:36605)
I20260927 10:53:13.678864 15178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:57007
I20260927 10:53:13.689430 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:53:13.700039 15217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:13.693036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.14.68.62:46619)
0927 10:53:13.693186 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:13.693189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:13.693290 (+ 101us) server_negotiation.cc:407] Connection header received
0927 10:53:13.693391 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:13.693394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:13.693435 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:13.693505 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:13.694193 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.694732 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.695535 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.695804 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.695915 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:13.697177 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:13.697180 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:13.697182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:13.697204 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:13.698538 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:13.698953 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:13.698956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:13.698957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:13.699033 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:13.699277 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:13.699279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:13.699280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:13.699496 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:13.699595 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:13.699833 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 10:53:13.699927 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 10:53:13.704936 15178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:53:13.708201 15178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:13.723260 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:13.716707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:45625 (local address 127.0.0.1:49364)
0927 10:53:13.717602 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:13.717657 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:13.717930 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:13.718577 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:13.718580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:13.718597 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:13.718745 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:13.718754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.720294 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.720298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:13.721517 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:13.721526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.722693 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.722697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:13.722803 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:13.722843 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 10:53:13.722959 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 10:53:13.723085 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":60,"thread_start_us":96,"threads_started":1}
I20260927 10:53:13.723287 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:13.716815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49364 (local address 127.14.68.2:45625)
0927 10:53:13.718157 (+ 1342us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:13.718173 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:13.718198 (+ 25us) server_negotiation.cc:407] Connection header received
0927 10:53:13.718280 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:13.718285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:13.718443 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:13.718599 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:13.718871 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.720123 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.721697 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.722564 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.722678 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:13.722856 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:13.723005 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 10:53:13.723102 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":99,"threads_started":1}
I20260927 10:53:13.723856 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:13.717110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47756 (local address 127.14.68.3:36605)
0927 10:53:13.717560 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:13.717572 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:13.718097 (+ 525us) server_negotiation.cc:407] Connection header received
0927 10:53:13.718186 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:13.718190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:13.718355 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:13.718517 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:13.718964 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.720135 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.721856 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.722667 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.722755 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:13.722862 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:13.723575 (+ 713us) server_negotiation.cc:299] Negotiation successful
0927 10:53:13.723680 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":130,"threads_started":1}
I20260927 10:53:13.723990 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:13.717467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:33213 (local address 127.0.0.1:50738)
0927 10:53:13.718064 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:13.718078 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:13.718172 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:13.719222 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:13.719226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:13.719243 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:13.719329 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:13.719337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.721096 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.721099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:13.722271 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:13.722279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.723581 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.723585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:13.723675 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:13.723730 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 10:53:13.723796 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 10:53:13.723864 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":3,"thread_start_us":89,"threads_started":1}
I20260927 10:53:13.724354 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:13.716719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:36605 (local address 127.0.0.1:47756)
0927 10:53:13.717918 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:13.717931 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:13.718021 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:13.718517 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:13.718523 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:13.718545 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:13.718787 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:13.718796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.720262 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.720265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:13.721752 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:13.721762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.722960 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.722965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:13.723076 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:13.723146 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 10:53:13.724137 (+ 991us) client_negotiation.cc:240] Negotiation successful
0927 10:53:13.724217 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1133,"mutex_wait_us":105,"spinlock_wait_cycles":16256,"thread_start_us":113,"threads_started":1}
I20260927 10:53:13.724447 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:13.717873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50738 (local address 127.14.68.1:33213)
0927 10:53:13.718820 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:13.718837 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:13.718860 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:53:13.718936 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:13.718940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:13.719081 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:13.719232 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:13.719456 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.720950 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.722377 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.723188 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.723371 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:13.723521 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:13.724202 (+ 681us) server_negotiation.cc:299] Negotiation successful
0927 10:53:13.724296 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":844,"thread_start_us":120,"threads_started":1}
I20260927 10:53:13.726856 15437 tablet_service.cc:1467] Processing CreateTablet for tablet 263ce8bfbb954567ae89a0e179d625cc (DEFAULT_TABLE table=test-table [id=85252818fc7f4781abff5232482fb6e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:13.727115 15571 tablet_service.cc:1467] Processing CreateTablet for tablet 263ce8bfbb954567ae89a0e179d625cc (DEFAULT_TABLE table=test-table [id=85252818fc7f4781abff5232482fb6e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:13.727115 15303 tablet_service.cc:1467] Processing CreateTablet for tablet 263ce8bfbb954567ae89a0e179d625cc (DEFAULT_TABLE table=test-table [id=85252818fc7f4781abff5232482fb6e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:13.728206 15303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 263ce8bfbb954567ae89a0e179d625cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:13.728206 15571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 263ce8bfbb954567ae89a0e179d625cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:13.728332 15437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 263ce8bfbb954567ae89a0e179d625cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:13.735435 15654 tablet_bootstrap.cc:492] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf: Bootstrap starting.
I20260927 10:53:13.736472 15655 tablet_bootstrap.cc:492] T 263ce8bfbb954567ae89a0e179d625cc P 9650258f3b214454ae28cc4c1ed0fced: Bootstrap starting.
I20260927 10:53:13.736847 15656 tablet_bootstrap.cc:492] T 263ce8bfbb954567ae89a0e179d625cc P 45ad276ff8304f45b5122bd1887a1a39: Bootstrap starting.
I20260927 10:53:13.738536 15654 tablet_bootstrap.cc:654] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:13.738626 15655 tablet_bootstrap.cc:654] T 263ce8bfbb954567ae89a0e179d625cc P 9650258f3b214454ae28cc4c1ed0fced: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:13.739346 15656 tablet_bootstrap.cc:654] T 263ce8bfbb954567ae89a0e179d625cc P 45ad276ff8304f45b5122bd1887a1a39: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:13.739499 15654 log.cc:824] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:13.739769 15655 log.cc:824] T 263ce8bfbb954567ae89a0e179d625cc P 9650258f3b214454ae28cc4c1ed0fced: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:13.740250 15656 log.cc:824] T 263ce8bfbb954567ae89a0e179d625cc P 45ad276ff8304f45b5122bd1887a1a39: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:13.741824 15655 tablet_bootstrap.cc:492] T 263ce8bfbb954567ae89a0e179d625cc P 9650258f3b214454ae28cc4c1ed0fced: No bootstrap required, opened a new log
I20260927 10:53:13.742017 15655 ts_tablet_manager.cc:1397] T 263ce8bfbb954567ae89a0e179d625cc P 9650258f3b214454ae28cc4c1ed0fced: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:53:13.742173 15656 tablet_bootstrap.cc:492] T 263ce8bfbb954567ae89a0e179d625cc P 45ad276ff8304f45b5122bd1887a1a39: No bootstrap required, opened a new log
I20260927 10:53:13.742236 15654 tablet_bootstrap.cc:492] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf: No bootstrap required, opened a new log
I20260927 10:53:13.742380 15656 ts_tablet_manager.cc:1397] T 263ce8bfbb954567ae89a0e179d625cc P 45ad276ff8304f45b5122bd1887a1a39: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:53:13.742424 15654 ts_tablet_manager.cc:1397] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 10:53:13.745235 15656 raft_consensus.cc:348] T 263ce8bfbb954567ae89a0e179d625cc P 45ad276ff8304f45b5122bd1887a1a39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ea75ab11d14c2e877d49c5a33645bf" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 45625 } } peers { permanent_uuid: "45ad276ff8304f45b5122bd1887a1a39" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36605 } } peers { permanent_uuid: "9650258f3b214454ae28cc4c1ed0fced" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33213 } }
I20260927 10:53:13.745507 15656 raft_consensus.cc:374] T 263ce8bfbb954567ae89a0e179d625cc P 45ad276ff8304f45b5122bd1887a1a39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:13.745579 15656 raft_consensus.cc:729] T 263ce8bfbb954567ae89a0e179d625cc P 45ad276ff8304f45b5122bd1887a1a39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45ad276ff8304f45b5122bd1887a1a39, State: Initialized, Role: FOLLOWER
I20260927 10:53:13.746070 15656 consensus_queue.cc:260] T 263ce8bfbb954567ae89a0e179d625cc P 45ad276ff8304f45b5122bd1887a1a39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ea75ab11d14c2e877d49c5a33645bf" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 45625 } } peers { permanent_uuid: "45ad276ff8304f45b5122bd1887a1a39" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36605 } } peers { permanent_uuid: "9650258f3b214454ae28cc4c1ed0fced" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33213 } }
I20260927 10:53:13.745951 15655 raft_consensus.cc:348] T 263ce8bfbb954567ae89a0e179d625cc P 9650258f3b214454ae28cc4c1ed0fced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ea75ab11d14c2e877d49c5a33645bf" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 45625 } } peers { permanent_uuid: "45ad276ff8304f45b5122bd1887a1a39" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36605 } } peers { permanent_uuid: "9650258f3b214454ae28cc4c1ed0fced" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33213 } }
I20260927 10:53:13.746070 15654 raft_consensus.cc:348] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ea75ab11d14c2e877d49c5a33645bf" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 45625 } } peers { permanent_uuid: "45ad276ff8304f45b5122bd1887a1a39" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36605 } } peers { permanent_uuid: "9650258f3b214454ae28cc4c1ed0fced" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33213 } }
I20260927 10:53:13.746306 15654 raft_consensus.cc:374] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:13.746306 15655 raft_consensus.cc:374] T 263ce8bfbb954567ae89a0e179d625cc P 9650258f3b214454ae28cc4c1ed0fced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:13.746367 15655 raft_consensus.cc:729] T 263ce8bfbb954567ae89a0e179d625cc P 9650258f3b214454ae28cc4c1ed0fced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9650258f3b214454ae28cc4c1ed0fced, State: Initialized, Role: FOLLOWER
I20260927 10:53:13.746367 15654 raft_consensus.cc:729] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ea75ab11d14c2e877d49c5a33645bf, State: Initialized, Role: FOLLOWER
I20260927 10:53:13.746809 15654 consensus_queue.cc:260] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ea75ab11d14c2e877d49c5a33645bf" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 45625 } } peers { permanent_uuid: "45ad276ff8304f45b5122bd1887a1a39" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36605 } } peers { permanent_uuid: "9650258f3b214454ae28cc4c1ed0fced" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33213 } }
I20260927 10:53:13.746809 15655 consensus_queue.cc:260] T 263ce8bfbb954567ae89a0e179d625cc P 9650258f3b214454ae28cc4c1ed0fced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ea75ab11d14c2e877d49c5a33645bf" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 45625 } } peers { permanent_uuid: "45ad276ff8304f45b5122bd1887a1a39" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36605 } } peers { permanent_uuid: "9650258f3b214454ae28cc4c1ed0fced" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33213 } }
I20260927 10:53:13.747131 15656 ts_tablet_manager.cc:1428] T 263ce8bfbb954567ae89a0e179d625cc P 45ad276ff8304f45b5122bd1887a1a39: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260927 10:53:13.747495 15637 heartbeater.cc:503] Master 127.14.68.62:46619 was elected leader, sending a full tablet report...
I20260927 10:53:13.747777 15655 ts_tablet_manager.cc:1428] T 263ce8bfbb954567ae89a0e179d625cc P 9650258f3b214454ae28cc4c1ed0fced: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:53:13.748102 15369 heartbeater.cc:503] Master 127.14.68.62:46619 was elected leader, sending a full tablet report...
I20260927 10:53:13.748631 15503 heartbeater.cc:503] Master 127.14.68.62:46619 was elected leader, sending a full tablet report...
I20260927 10:53:13.748868 15654 ts_tablet_manager.cc:1428] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260927 10:53:13.793867 15662 raft_consensus.cc:482] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:13.794065 15662 raft_consensus.cc:504] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ea75ab11d14c2e877d49c5a33645bf" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 45625 } } peers { permanent_uuid: "45ad276ff8304f45b5122bd1887a1a39" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36605 } } peers { permanent_uuid: "9650258f3b214454ae28cc4c1ed0fced" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33213 } }
I20260927 10:53:13.795267 15662 leader_election.cc:288] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45ad276ff8304f45b5122bd1887a1a39 (127.14.68.3:36605), 9650258f3b214454ae28cc4c1ed0fced (127.14.68.1:33213)
I20260927 10:53:13.799535 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:13.795494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:36605 (local address 127.14.68.2:34587)
0927 10:53:13.795635 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:13.795653 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:13.795763 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:13.796236 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:13.796239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:13.796258 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:13.796379 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:13.796385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.797220 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.797223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:13.798367 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:13.798377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.799185 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.799187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:13.799267 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:13.799286 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 10:53:13.799347 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 10:53:13.799400 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 10:53:13.799599 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:13.795813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:34587 (local address 127.14.68.3:36605)
0927 10:53:13.795945 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:13.795949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:13.795960 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:53:13.796005 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:13.796007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:13.796048 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:13.796216 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:13.796489 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.797108 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.798503 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.799058 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.799176 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:13.799239 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:13.799386 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 10:53:13.799474 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:53:13.800038 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:13.795566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:38291 (local address 127.14.68.1:33213)
0927 10:53:13.795715 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:13.795720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:13.796337 (+ 617us) server_negotiation.cc:407] Connection header received
0927 10:53:13.796487 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:13.796490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:13.796541 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:13.796628 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:13.796939 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.797505 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.798837 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.799423 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.799524 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:13.799596 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:13.799862 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 10:53:13.799934 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:53:13.800067 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:13.795494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:33213 (local address 127.14.68.2:38291)
0927 10:53:13.796233 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:13.796244 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:13.796361 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:13.796641 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:13.796644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:13.796659 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:13.796783 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:13.796808 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.797625 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.797628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:13.798711 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:13.798718 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.799570 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.799576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:13.799674 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:13.799723 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 10:53:13.799840 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 10:53:13.799904 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":340,"thread_start_us":150,"threads_started":1}
I20260927 10:53:13.800241 15591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263ce8bfbb954567ae89a0e179d625cc" candidate_uuid: "36ea75ab11d14c2e877d49c5a33645bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45ad276ff8304f45b5122bd1887a1a39" is_pre_election: true
I20260927 10:53:13.800612 15591 raft_consensus.cc:2454] T 263ce8bfbb954567ae89a0e179d625cc P 45ad276ff8304f45b5122bd1887a1a39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36ea75ab11d14c2e877d49c5a33645bf in term 0.
I20260927 10:53:13.800683 15323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263ce8bfbb954567ae89a0e179d625cc" candidate_uuid: "36ea75ab11d14c2e877d49c5a33645bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9650258f3b214454ae28cc4c1ed0fced" is_pre_election: true
I20260927 10:53:13.800952 15323 raft_consensus.cc:2454] T 263ce8bfbb954567ae89a0e179d625cc P 9650258f3b214454ae28cc4c1ed0fced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36ea75ab11d14c2e877d49c5a33645bf in term 0.
I20260927 10:53:13.801083 15393 leader_election.cc:302] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ea75ab11d14c2e877d49c5a33645bf, 45ad276ff8304f45b5122bd1887a1a39; no voters:
I20260927 10:53:13.801416 15662 raft_consensus.cc:2781] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:13.801505 15662 raft_consensus.cc:482] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:13.801548 15662 raft_consensus.cc:3037] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:13.802682 15662 raft_consensus.cc:504] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ea75ab11d14c2e877d49c5a33645bf" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 45625 } } peers { permanent_uuid: "45ad276ff8304f45b5122bd1887a1a39" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36605 } } peers { permanent_uuid: "9650258f3b214454ae28cc4c1ed0fced" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33213 } }
I20260927 10:53:13.803028 15662 leader_election.cc:288] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf [CANDIDATE]: Term 1 election: Requested vote from peers 45ad276ff8304f45b5122bd1887a1a39 (127.14.68.3:36605), 9650258f3b214454ae28cc4c1ed0fced (127.14.68.1:33213)
I20260927 10:53:13.803370 15323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263ce8bfbb954567ae89a0e179d625cc" candidate_uuid: "36ea75ab11d14c2e877d49c5a33645bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9650258f3b214454ae28cc4c1ed0fced"
I20260927 10:53:13.803370 15591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263ce8bfbb954567ae89a0e179d625cc" candidate_uuid: "36ea75ab11d14c2e877d49c5a33645bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45ad276ff8304f45b5122bd1887a1a39"
I20260927 10:53:13.803493 15323 raft_consensus.cc:3037] T 263ce8bfbb954567ae89a0e179d625cc P 9650258f3b214454ae28cc4c1ed0fced [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:13.803493 15591 raft_consensus.cc:3037] T 263ce8bfbb954567ae89a0e179d625cc P 45ad276ff8304f45b5122bd1887a1a39 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:13.804806 15591 raft_consensus.cc:2454] T 263ce8bfbb954567ae89a0e179d625cc P 45ad276ff8304f45b5122bd1887a1a39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ea75ab11d14c2e877d49c5a33645bf in term 1.
I20260927 10:53:13.804811 15323 raft_consensus.cc:2454] T 263ce8bfbb954567ae89a0e179d625cc P 9650258f3b214454ae28cc4c1ed0fced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ea75ab11d14c2e877d49c5a33645bf in term 1.
I20260927 10:53:13.805107 15393 leader_election.cc:302] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ea75ab11d14c2e877d49c5a33645bf, 45ad276ff8304f45b5122bd1887a1a39; no voters:
I20260927 10:53:13.805289 15662 raft_consensus.cc:2781] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:13.805532 15662 raft_consensus.cc:686] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf [term 1 LEADER]: Becoming Leader. State: Replica: 36ea75ab11d14c2e877d49c5a33645bf, State: Running, Role: LEADER
I20260927 10:53:13.806064 15662 consensus_queue.cc:237] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ea75ab11d14c2e877d49c5a33645bf" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 45625 } } peers { permanent_uuid: "45ad276ff8304f45b5122bd1887a1a39" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36605 } } peers { permanent_uuid: "9650258f3b214454ae28cc4c1ed0fced" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33213 } }
I20260927 10:53:13.809376 15177 catalog_manager.cc:5495] T 263ce8bfbb954567ae89a0e179d625cc P 36ea75ab11d14c2e877d49c5a33645bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 36ea75ab11d14c2e877d49c5a33645bf (127.14.68.2). New cstate: current_term: 1 leader_uuid: "36ea75ab11d14c2e877d49c5a33645bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ea75ab11d14c2e877d49c5a33645bf" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 45625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45ad276ff8304f45b5122bd1887a1a39" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9650258f3b214454ae28cc4c1ed0fced" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33213 } health_report { overall_health: UNKNOWN } } }
Sep 27 10:53:13 dist-test-slave-tz51 krb5kdc[15133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.1@KRBTEST.COM
I20260927 10:53:13.830118 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:13.816743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50740 (local address 127.14.68.1:33213)
0927 10:53:13.819048 (+ 2305us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:13.819052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:13.819066 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:53:13.819127 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:13.819130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:13.819179 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:13.819252 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:13.820278 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.821022 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.822209 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.822446 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.822565 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:13.825770 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:13.825790 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:13.825812 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:13.825845 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:13.828122 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:13.828639 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:13.828644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:13.828646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:13.828769 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:13.829106 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:13.829109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:13.829111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:13.829380 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:13.829606 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:13.829813 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 10:53:13.829934 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2207}
Sep 27 10:53:13 dist-test-slave-tz51 krb5kdc[15133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506393, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:53:13 dist-test-slave-tz51 krb5kdc[15133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506393, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.68.1@KRBTEST.COM
I20260927 10:53:13.858101 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:13.849652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50750 (local address 127.14.68.1:33213)
0927 10:53:13.849797 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:13.849800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:13.849949 (+ 149us) server_negotiation.cc:407] Connection header received
0927 10:53:13.850063 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:13.850066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:13.850107 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:13.850185 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:13.850835 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.851365 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.852336 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.852558 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.852637 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:13.854958 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:13.854961 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:13.854963 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:13.854985 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:13.856439 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:13.856861 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:13.856864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:13.856866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:13.856946 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:13.857214 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:13.857220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:13.857225 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:13.857390 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:13.857523 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:13.857813 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 10:53:13.857939 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 10:53:13 dist-test-slave-tz51 krb5kdc[15133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:53:13 dist-test-slave-tz51 krb5kdc[15133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.1@KRBTEST.COM
I20260927 10:53:13.883827 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:13.875082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50760 (local address 127.14.68.1:33213)
0927 10:53:13.875225 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:13.875247 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:13.875477 (+ 230us) server_negotiation.cc:407] Connection header received
0927 10:53:13.875647 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:13.875650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:13.875758 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:13.875857 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:13.876568 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.877090 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.877933 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:13.878160 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:13.878271 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:13.880729 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:13.880733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:13.880738 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:13.880766 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:13.882155 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:13.882575 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:13.882578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:13.882580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:13.882661 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:13.882938 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:13.882942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:13.882944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:13.883118 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:13.883263 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:13.883491 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 10:53:13.883611 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 10:53:13.885249 15283 tablet_service.cc:2908] Rejecting scan request for tablet 263ce8bfbb954567ae89a0e179d625cc: Uninitialized: clean time has not yet been initialized
I20260927 10:53:13.887300 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 15240
I20260927 10:53:13.893185 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 15374
I20260927 10:53:13.898840 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 15508
I20260927 10:53:13.904357 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 15146
2026-09-27T10:53:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2299 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:53:13 dist-test-slave-tz51 krb5kdc[15678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:53:13 dist-test-slave-tz51 krb5kdc[15678](info): set up 2 sockets
Sep 27 10:53:13 dist-test-slave-tz51 krb5kdc[15678](info): commencing operation
krb5kdc: starting...
W20260927 10:53:15.043264 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:53:15 dist-test-slave-tz51 krb5kdc[15678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:53:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:53:15.166343 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:43327
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:46781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:43327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:15.274430 15691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:15.274685 15691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:15.274724 15691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:15.278657 15691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:15.278712 15691 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 10:53:15.278728 15691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:15.278746 15691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:15.278762 15691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:15.278777 15691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:15.284379 15691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:46781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:43327
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:43327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:15.285398 15691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:15.286397 15691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:15.294965 15697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:15.294961 15696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:15.295205 15691 server_base.cc:1034] running on GCE node
W20260927 10:53:15.294956 15699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:15.295969 15691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:15.296988 15691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:15.298156 15691 hybrid_clock.cc:648] HybridClock initialized: now 1790506395298142 us; error 26 us; skew 500 ppm
Sep 27 10:53:15 dist-test-slave-tz51 krb5kdc[15678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506395, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:15.307009 15691 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:53:15.308056 15691 webserver.cc:460] Webserver started at http://127.14.68.62:43645/ using document root <none> and password file <none>
I20260927 10:53:15.308638 15691 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:15.308702 15691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:15.308923 15691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:15.310964 15691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "4a5e7f5ea3d24567adf8f3b5b272e541"
format_stamp: "Formatted at 2026-09-27 10:53:15 on dist-test-slave-tz51"
server_key: "3ebaa69655fc454d7e071f4d3ef2c5ff"
server_key_iv: "092f7db624358081e4abf5c1a906b0c9"
server_key_version: "encryptionkey@0"
I20260927 10:53:15.311570 15691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "4a5e7f5ea3d24567adf8f3b5b272e541"
format_stamp: "Formatted at 2026-09-27 10:53:15 on dist-test-slave-tz51"
server_key: "3ebaa69655fc454d7e071f4d3ef2c5ff"
server_key_iv: "092f7db624358081e4abf5c1a906b0c9"
server_key_version: "encryptionkey@0"
I20260927 10:53:15.315245 15691 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:53:15.317732 15706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:15.319027 15691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:15.319175 15691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "4a5e7f5ea3d24567adf8f3b5b272e541"
format_stamp: "Formatted at 2026-09-27 10:53:15 on dist-test-slave-tz51"
server_key: "3ebaa69655fc454d7e071f4d3ef2c5ff"
server_key_iv: "092f7db624358081e4abf5c1a906b0c9"
server_key_version: "encryptionkey@0"
I20260927 10:53:15.319288 15691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:15.339077 15691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:15.342074 15691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:15.342270 15691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:15.350186 15691 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:43327
I20260927 10:53:15.350210 15768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:43327 every 8 connection(s)
I20260927 10:53:15.351087 15691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:15.352208 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 15691
I20260927 10:53:15.352346 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:53:15.352617 14608 external_mini_cluster.cc:1428] Setting key 14908cbc7fd66f67542d356714d8efd5
I20260927 10:53:15.354148 15769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:15.360715 15769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541: Bootstrap starting.
Sep 27 10:53:15 dist-test-slave-tz51 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:15.363022 15769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:15.364003 15769 log.cc:824] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:15.366204 15769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541: No bootstrap required, opened a new log
I20260927 10:53:15.367928 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:15.353961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40984 (local address 127.14.68.62:43327)
0927 10:53:15.354369 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:15.354387 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:15.354426 (+ 39us) server_negotiation.cc:407] Connection header received
0927 10:53:15.355220 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:15.355242 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:15.355652 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:15.356028 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:15.356684 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:15.358175 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:15.358984 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:15.359660 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:15.359847 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:15.362368 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:15.362373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:15.362388 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:15.362416 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:15.364943 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:15.365381 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:15.365387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:15.365403 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:15.365529 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:15.365798 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:15.365801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:15.365803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:15.366316 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:15.366482 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:15.366733 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 10:53:15.367003 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":107,"threads_started":1}
I20260927 10:53:15.369719 15769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5e7f5ea3d24567adf8f3b5b272e541" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43327 } }
I20260927 10:53:15.370014 15769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:15.370076 15769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a5e7f5ea3d24567adf8f3b5b272e541, State: Initialized, Role: FOLLOWER
I20260927 10:53:15.370508 15769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5e7f5ea3d24567adf8f3b5b272e541" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43327 } }
I20260927 10:53:15.370664 15769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:15.370729 15769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:15.370822 15769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:15.371613 15769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5e7f5ea3d24567adf8f3b5b272e541" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43327 } }
I20260927 10:53:15.371948 15769 leader_election.cc:302] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a5e7f5ea3d24567adf8f3b5b272e541; no voters:
I20260927 10:53:15.372262 15769 leader_election.cc:288] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:15.372354 15774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:15.372606 15774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541 [term 1 LEADER]: Becoming Leader. State: Replica: 4a5e7f5ea3d24567adf8f3b5b272e541, State: Running, Role: LEADER
I20260927 10:53:15.372918 15774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5e7f5ea3d24567adf8f3b5b272e541" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43327 } }
I20260927 10:53:15.373409 15769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:15.374540 15776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a5e7f5ea3d24567adf8f3b5b272e541. Latest consensus state: current_term: 1 leader_uuid: "4a5e7f5ea3d24567adf8f3b5b272e541" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5e7f5ea3d24567adf8f3b5b272e541" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43327 } } }
I20260927 10:53:15.374975 15776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:15.374872 15775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a5e7f5ea3d24567adf8f3b5b272e541" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5e7f5ea3d24567adf8f3b5b272e541" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43327 } } }
I20260927 10:53:15.375319 15775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:15.375411 15785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:15.378612 15785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:15.383793 15785 catalog_manager.cc:1347] Generated new cluster ID: c763bf75690543e3bb0924340dec468f
I20260927 10:53:15.383858 15785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:15.398857 15785 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:15.399745 15785 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:15.405131 15785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541: Generated new TSK 0
I20260927 10:53:15.405714 15785 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
I20260927 10:53:15.435037 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:43327
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:46781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:15.543979 15795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:15.544232 15795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:15.544277 15795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:15.548329 15795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:15.548384 15795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:15.548421 15795 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:53:15.548473 15795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:53:15.553714 15795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:46781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.68.62:43327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:15.554714 15795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:15.555779 15795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:15.565114 15801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:15.565099 15800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:15.565099 15803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:15.565270 15795 server_base.cc:1034] running on GCE node
I20260927 10:53:15.565765 15795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:15.566419 15795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:15.567577 15795 hybrid_clock.cc:648] HybridClock initialized: now 1790506395567567 us; error 29 us; skew 500 ppm
Sep 27 10:53:15 dist-test-slave-tz51 krb5kdc[15678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506395, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:15.570291 15795 init.cc:377] Logged in from keytab as kudu/127.14.68.1@KRBTEST.COM (short username kudu)
I20260927 10:53:15.571087 15795 webserver.cc:460] Webserver started at http://127.14.68.1:43943/ using document root <none> and password file <none>
I20260927 10:53:15.571568 15795 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:15.571625 15795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:15.571846 15795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:15.573869 15795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "fa95d82fe8da48a7a5802d9a197cb479"
format_stamp: "Formatted at 2026-09-27 10:53:15 on dist-test-slave-tz51"
server_key: "d8ca95d77d78b5ceea6be03324b3ecf3"
server_key_iv: "63ee4959f17d61d1bd8fda0c9cbe8a8f"
server_key_version: "encryptionkey@0"
I20260927 10:53:15.574371 15795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "fa95d82fe8da48a7a5802d9a197cb479"
format_stamp: "Formatted at 2026-09-27 10:53:15 on dist-test-slave-tz51"
server_key: "d8ca95d77d78b5ceea6be03324b3ecf3"
server_key_iv: "63ee4959f17d61d1bd8fda0c9cbe8a8f"
server_key_version: "encryptionkey@0"
I20260927 10:53:15.578203 15795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:53:15.580885 15810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:15.582043 15795 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:15.582144 15795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "fa95d82fe8da48a7a5802d9a197cb479"
format_stamp: "Formatted at 2026-09-27 10:53:15 on dist-test-slave-tz51"
server_key: "d8ca95d77d78b5ceea6be03324b3ecf3"
server_key_iv: "63ee4959f17d61d1bd8fda0c9cbe8a8f"
server_key_version: "encryptionkey@0"
I20260927 10:53:15.582275 15795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:15.596652 15795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:15.599579 15795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:15.599818 15795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:15.601334 15795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:15.601380 15795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:15.601419 15795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:15.601471 15795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:53:15 dist-test-slave-tz51 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506395, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:15.612587 15795 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:37477
I20260927 10:53:15.612645 15925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:37477 every 8 connection(s)
I20260927 10:53:15.613466 15795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:53:15.615867 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:15.602412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:44763 (local address 127.14.68.62:43327)
0927 10:53:15.602560 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:15.602565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:15.603603 (+ 1038us) server_negotiation.cc:407] Connection header received
0927 10:53:15.604800 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:15.604804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:15.604865 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:15.604985 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:15.606646 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:15.607560 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:15.608931 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:15.609190 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:15.609306 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:15.611962 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:15.611966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:15.611969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:15.611998 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:15.613763 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:15.614374 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:15.614377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:15.614380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:15.614479 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:15.614799 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:15.614801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:15.614802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:15.614974 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:15.615048 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:15.615522 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 10:53:15.615643 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:53:15.616706 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:15.602948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:43327 (local address 127.14.68.1:44763)
0927 10:53:15.603435 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:15.603477 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:15.604539 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:15.605214 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:15.605224 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:15.605710 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:15.606465 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:15.606490 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:15.607681 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:15.607685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:15.608808 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:15.608818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:15.609346 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:15.609350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:15.609529 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:15.610138 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:15.610156 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:15.611785 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:15.613884 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:15.613891 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:15.613906 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:15.614236 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:15.614576 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:15.614578 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:15.614580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:15.614710 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:15.615062 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:15.615067 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:15.615316 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 10:53:15.615461 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 10:53:15.615789 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":136,"threads_started":1}
I20260927 10:53:15.618089 15926 heartbeater.cc:347] Connected to a master server at 127.14.68.62:43327
I20260927 10:53:15.618251 15926 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:15.618777 15926 heartbeater.cc:511] Master 127.14.68.62:43327 requested a full tablet report, sending...
I20260927 10:53:15.620432 15722 ts_manager.cc:194] Registered new tserver with Master: fa95d82fe8da48a7a5802d9a197cb479 (127.14.68.1:37477)
I20260927 10:53:15.621362 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 15795
I20260927 10:53:15.621474 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:53:15.621737 14608 external_mini_cluster.cc:1428] Setting key f2e0bffd57529fe4c041ca190e99c6d9
I20260927 10:53:15.621982 15722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:44763
I20260927 10:53:15.628918 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:15.620264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:50437 (local address 127.14.68.62:43327)
0927 10:53:15.620391 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:15.620397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:15.620638 (+ 241us) server_negotiation.cc:407] Connection header received
0927 10:53:15.620740 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:15.620742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:15.620785 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:15.620871 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:15.621242 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:15.621846 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:15.622907 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:15.623199 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:15.623260 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:15.625773 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:15.625777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:15.625779 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:15.625802 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:15.627188 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:15.627743 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:15.627752 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:15.627754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:15.627825 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:15.628156 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:15.628159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:15.628160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:15.628317 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:15.628391 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:15.628717 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 10:53:15.628805 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":50}
I20260927 10:53:15.629117 15932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:15.620203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:43327 (local address 127.14.68.1:50437)
0927 10:53:15.620526 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:15.620559 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:15.620672 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:15.620876 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:15.620878 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:15.621039 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:15.621131 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:15.621136 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:15.621969 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:15.621973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:15.622791 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:15.622803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:15.624572 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:15.624579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:15.624717 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:15.625289 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:15.625309 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:15.625634 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:15.627300 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:15.627306 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:15.627309 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:15.627591 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:15.627919 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:15.627922 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:15.627924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:15.628033 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:15.628416 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:15.628422 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:15.628534 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 10:53:15.628674 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 10:53:15.628841 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.14.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
I20260927 10:53:15.650362 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:43327
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:46781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:15.758226 15935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:15.758452 15935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:15.758491 15935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:15.762424 15935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:15.762475 15935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:15.762512 15935 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:53:15.762544 15935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:53:15.767433 15935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:46781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.68.62:43327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:15.768412 15935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:15.769311 15935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:15.778828 15941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:15.778867 15935 server_base.cc:1034] running on GCE node
W20260927 10:53:15.778880 15940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:15.779021 15943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:15.779500 15935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:15.780176 15935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:15.781363 15935 hybrid_clock.cc:648] HybridClock initialized: now 1790506395781349 us; error 26 us; skew 500 ppm
Sep 27 10:53:15 dist-test-slave-tz51 krb5kdc[15678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506395, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:15.783973 15935 init.cc:377] Logged in from keytab as kudu/127.14.68.2@KRBTEST.COM (short username kudu)
I20260927 10:53:15.784749 15935 webserver.cc:460] Webserver started at http://127.14.68.2:40831/ using document root <none> and password file <none>
I20260927 10:53:15.785394 15935 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:15.785480 15935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:15.785712 15935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:15.787689 15935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "21a6a8d06f2648a09dd95aef8b9f5968"
format_stamp: "Formatted at 2026-09-27 10:53:15 on dist-test-slave-tz51"
server_key: "5bec4359cec4a62545033a0be01836c4"
server_key_iv: "43c362928c7bc39d30b448fdb6b06ca1"
server_key_version: "encryptionkey@0"
I20260927 10:53:15.788286 15935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "21a6a8d06f2648a09dd95aef8b9f5968"
format_stamp: "Formatted at 2026-09-27 10:53:15 on dist-test-slave-tz51"
server_key: "5bec4359cec4a62545033a0be01836c4"
server_key_iv: "43c362928c7bc39d30b448fdb6b06ca1"
server_key_version: "encryptionkey@0"
I20260927 10:53:15.791770 15935 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:53:15.794162 15950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:15.795276 15935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:53:15.795414 15935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "21a6a8d06f2648a09dd95aef8b9f5968"
format_stamp: "Formatted at 2026-09-27 10:53:15 on dist-test-slave-tz51"
server_key: "5bec4359cec4a62545033a0be01836c4"
server_key_iv: "43c362928c7bc39d30b448fdb6b06ca1"
server_key_version: "encryptionkey@0"
I20260927 10:53:15.795527 15935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:15.818178 15935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:15.821311 15935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:15.821518 15935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:15.823139 15935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:15.823207 15935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:15.823272 15935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:15.823311 15935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:53:15 dist-test-slave-tz51 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506395, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:15.834110 15935 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:32981
I20260927 10:53:15.834295 16065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:32981 every 8 connection(s)
I20260927 10:53:15.835238 15935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:53:15.836364 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 15935
I20260927 10:53:15.836499 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:53:15.836802 14608 external_mini_cluster.cc:1428] Setting key 71c66973e4ee8c0f6f291021ca321cee
I20260927 10:53:15.837350 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:15.823895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:51077 (local address 127.14.68.62:43327)
0927 10:53:15.824016 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:15.824021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:15.824986 (+ 965us) server_negotiation.cc:407] Connection header received
0927 10:53:15.826029 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:15.826034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:15.826115 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:15.826184 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:15.827970 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:15.828694 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:15.829885 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:15.830241 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:15.830355 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:15.833318 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:15.833321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:15.833323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:15.833345 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:15.835003 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:15.835621 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:15.835628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:15.835633 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:15.835753 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:15.836169 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:15.836172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:15.836173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:15.836346 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:15.836424 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:15.837010 (+ 586us) server_negotiation.cc:299] Negotiation successful
0927 10:53:15.837182 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 10:53:15.838239 15959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:15.824386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:43327 (local address 127.14.68.2:51077)
0927 10:53:15.824823 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:15.824870 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:15.825785 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:15.826645 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:15.826654 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:15.827033 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:15.827810 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:15.827833 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:15.828803 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:15.828807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:15.829767 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:15.829775 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:15.830442 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:15.830448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:15.830614 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:15.831174 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:15.831193 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:15.833135 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:15.835172 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:15.835180 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:15.835195 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:15.835495 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:15.835879 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:15.835886 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:15.835888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:15.836060 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:15.836460 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:15.836466 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:15.836761 (+ 295us) client_negotiation.cc:769] Sending connection context
0927 10:53:15.836957 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 10:53:15.837284 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":126,"threads_started":1}
I20260927 10:53:15.839639 16066 heartbeater.cc:347] Connected to a master server at 127.14.68.62:43327
I20260927 10:53:15.839803 16066 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:15.840369 16066 heartbeater.cc:511] Master 127.14.68.62:43327 requested a full tablet report, sending...
I20260927 10:53:15.841461 15722 ts_manager.cc:194] Registered new tserver with Master: 21a6a8d06f2648a09dd95aef8b9f5968 (127.14.68.2:32981)
I20260927 10:53:15.842157 15722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:51077
WARNING: no policy specified for kudu/127.14.68.3@KRBTEST.COM; defaulting to no policy
I20260927 10:53:15.849519 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:15.841943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:47623 (local address 127.14.68.62:43327)
0927 10:53:15.842079 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:15.842083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:15.842316 (+ 233us) server_negotiation.cc:407] Connection header received
0927 10:53:15.842458 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:15.842464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:15.842518 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:15.842604 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:15.843059 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:15.843600 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:15.844651 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:15.845072 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:15.845216 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:15.846306 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:15.846311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:15.846314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:15.846341 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:15.847991 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:15.848390 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:15.848393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:15.848395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:15.848482 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:15.848780 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:15.848785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:15.848787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:15.849029 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:15.849112 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:15.849317 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 10:53:15.849408 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:53:15.849551 16073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:15.841868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:43327 (local address 127.14.68.2:47623)
0927 10:53:15.842242 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:15.842258 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:15.842355 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:15.842637 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:15.842640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:15.842845 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:15.842944 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:15.842950 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:15.843753 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:15.843757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:15.844474 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:15.844485 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:15.845205 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:15.845207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:15.845279 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:15.845802 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:15.845813 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:15.846160 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:15.848097 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:15.848099 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:15.848100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:15.848296 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:15.848592 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:15.848595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:15.848599 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:15.848687 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:15.849133 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:15.849135 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:15.849206 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 10:53:15.849280 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 10:53:15.849383 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
I20260927 10:53:15.866443 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:43327
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:46781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:15.976570 16075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:15.976821 16075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:15.976898 16075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:15.980789 16075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:15.980863 16075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:15.980931 16075 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:53:15.981019 16075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:53:15.986711 16075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:46781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.68.62:43327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:15.987802 16075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:15.988835 16075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:15.998720 16080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:15.998720 16075 server_base.cc:1034] running on GCE node
W20260927 10:53:15.998709 16081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:15.998709 16083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:15.999398 16075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:16.000092 16075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:16.001308 16075 hybrid_clock.cc:648] HybridClock initialized: now 1790506396001293 us; error 28 us; skew 500 ppm
Sep 27 10:53:15 dist-test-slave-tz51 krb5kdc[15678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506395, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:16.003949 16075 init.cc:377] Logged in from keytab as kudu/127.14.68.3@KRBTEST.COM (short username kudu)
I20260927 10:53:16.004774 16075 webserver.cc:460] Webserver started at http://127.14.68.3:37791/ using document root <none> and password file <none>
I20260927 10:53:16.005319 16075 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:16.005410 16075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:16.005646 16075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:16.007669 16075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "1df8519da8b747b48e64cbee84667d83"
format_stamp: "Formatted at 2026-09-27 10:53:16 on dist-test-slave-tz51"
server_key: "78d523ab9de652c315d5a18d9a9f6fc7"
server_key_iv: "60ac5a5200d49e8636ea466613c9df88"
server_key_version: "encryptionkey@0"
I20260927 10:53:16.008282 16075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "1df8519da8b747b48e64cbee84667d83"
format_stamp: "Formatted at 2026-09-27 10:53:16 on dist-test-slave-tz51"
server_key: "78d523ab9de652c315d5a18d9a9f6fc7"
server_key_iv: "60ac5a5200d49e8636ea466613c9df88"
server_key_version: "encryptionkey@0"
I20260927 10:53:16.011873 16075 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:53:16.014350 16090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:16.015872 16075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:53:16.015962 16075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "1df8519da8b747b48e64cbee84667d83"
format_stamp: "Formatted at 2026-09-27 10:53:16 on dist-test-slave-tz51"
server_key: "78d523ab9de652c315d5a18d9a9f6fc7"
server_key_iv: "60ac5a5200d49e8636ea466613c9df88"
server_key_version: "encryptionkey@0"
I20260927 10:53:16.016090 16075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:16.028690 16075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:16.031766 16075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:16.031949 16075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:16.033267 16075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:16.033313 16075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:16.033350 16075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:16.033447 16075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:53:16 dist-test-slave-tz51 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506395, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:16.044267 16075 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:45303
I20260927 10:53:16.044351 16205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:45303 every 8 connection(s)
I20260927 10:53:16.045315 16075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:53:16.047108 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.034168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:43809 (local address 127.14.68.62:43327)
0927 10:53:16.034298 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:16.034302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:16.035294 (+ 992us) server_negotiation.cc:407] Connection header received
0927 10:53:16.036428 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:16.036432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:16.036503 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:16.036614 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:16.037847 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.038773 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.040460 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.040681 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.040795 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.043189 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:16.043193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:16.043196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:16.043223 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:16.044885 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:16.045442 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:16.045445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:16.045447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:16.045531 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:16.045969 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:16.045975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:16.045978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:16.046259 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:16.046375 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:16.046860 (+ 485us) server_negotiation.cc:299] Negotiation successful
0927 10:53:16.046969 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:53:16.047887 16099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.034672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:43327 (local address 127.14.68.3:43809)
0927 10:53:16.035148 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:16.035187 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:16.036166 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:16.036809 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:16.036836 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:16.037285 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:16.037719 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.037731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.038929 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.038932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.040341 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.040350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.040837 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.040841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.041002 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.041551 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:16.041566 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:16.043043 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:16.045019 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:16.045023 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:16.045034 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:16.045300 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:16.045628 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:16.045631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:16.045632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:16.045838 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:16.046382 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:16.046388 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:16.046642 (+ 254us) client_negotiation.cc:769] Sending connection context
0927 10:53:16.046793 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 10:53:16.047009 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":144,"threads_started":1}
I20260927 10:53:16.048990 16206 heartbeater.cc:347] Connected to a master server at 127.14.68.62:43327
I20260927 10:53:16.049151 16206 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:16.049661 16206 heartbeater.cc:511] Master 127.14.68.62:43327 requested a full tablet report, sending...
I20260927 10:53:16.050794 15723 ts_manager.cc:194] Registered new tserver with Master: 1df8519da8b747b48e64cbee84667d83 (127.14.68.3:45303)
I20260927 10:53:16.051575 15723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:43809
I20260927 10:53:16.052583 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 16075
I20260927 10:53:16.052695 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:53:16.053009 14608 external_mini_cluster.cc:1428] Setting key 52ff0981b7cc78e93fff8ba7b0b545ed
I20260927 10:53:16.054935 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:53:16.059209 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.051166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:43327 (local address 127.14.68.3:37589)
0927 10:53:16.051449 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:16.051463 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:16.051555 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:16.051856 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:16.051860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:16.052033 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:16.052121 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.052126 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.052912 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.052915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.053715 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.053726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.054761 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.054766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.054859 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.055409 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:16.055423 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:16.055765 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:16.057604 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:16.057609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:16.057613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:16.057876 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:16.058253 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:16.058255 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:16.058257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:16.058332 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:16.058809 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:16.058812 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:16.058888 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 10:53:16.058972 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 10:53:16.059075 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":114,"threads_started":1}
I20260927 10:53:16.059473 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.051224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:37589 (local address 127.14.68.62:43327)
0927 10:53:16.051349 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:16.051354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:16.051532 (+ 178us) server_negotiation.cc:407] Connection header received
0927 10:53:16.051664 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:16.051670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:16.051754 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:16.051815 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:16.052236 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.052793 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.053837 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.054162 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.054270 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.055900 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:16.055907 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:16.055913 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:16.055943 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:16.057483 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:16.058023 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:16.058030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:16.058036 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:16.058144 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:16.058431 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:16.058438 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:16.058443 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:16.058701 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:16.058816 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:16.059074 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 10:53:16.059215 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 10:53:16 dist-test-slave-tz51 krb5kdc[15678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506396, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:53:16 dist-test-slave-tz51 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506396, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:16.082491 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.074506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40988 (local address 127.14.68.62:43327)
0927 10:53:16.074657 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:16.074660 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:16.074672 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:53:16.074704 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:16.074706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:16.074741 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:16.074829 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:16.075500 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.076071 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.076900 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.077116 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.077220 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.079536 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:16.079544 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:16.079546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:16.079569 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:16.080890 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:16.081300 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:16.081305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:16.081307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:16.081385 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:16.081652 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:16.081656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:16.081657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:16.081828 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:16.081964 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:16.082213 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 10:53:16.082324 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:53:16.085243 15723 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:53:16.088469 15723 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:16.102376 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.096786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:37477 (local address 127.0.0.1:52246)
0927 10:53:16.097143 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:16.097156 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:16.097627 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:16.098195 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:16.098198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:16.098209 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:16.098352 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.098358 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.099544 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.099550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.101080 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.101090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.101862 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.101866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.101966 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.101999 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 10:53:16.102115 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 10:53:16.102231 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":5,"thread_start_us":93,"threads_started":1}
I20260927 10:53:16.102388 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.097440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52246 (local address 127.14.68.1:37477)
0927 10:53:16.097698 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:16.097716 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:16.097738 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:53:16.097809 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:16.097814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:16.098019 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:16.098504 (+ 485us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:16.098658 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.099441 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.101220 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.101739 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.101864 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.102034 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:16.102176 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 10:53:16.102265 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":94,"threads_started":1}
I20260927 10:53:16.102800 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.096414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:32981 (local address 127.0.0.1:35298)
0927 10:53:16.096752 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:16.096803 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:16.096928 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:16.098162 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:16.098170 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:16.098198 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:16.098512 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.098518 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.099921 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.099924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.101066 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.101076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.102122 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.102126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.102252 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.102274 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:53:16.102337 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 10:53:16.102421 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":257,"thread_start_us":95,"threads_started":1}
I20260927 10:53:16.103329 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.097138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:45303 (local address 127.0.0.1:53214)
0927 10:53:16.097348 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:16.097361 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:16.097448 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:16.098281 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:16.098283 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:16.098294 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:16.098358 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.098363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.099791 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.099794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.101032 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.101043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.102122 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.102126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.103024 (+ 898us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.103042 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 10:53:16.103140 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 10:53:16.103207 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":817,"thread_start_us":89,"threads_started":1}
I20260927 10:53:16.103605 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.097321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53214 (local address 127.14.68.3:45303)
0927 10:53:16.097858 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:16.097868 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:16.097886 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:53:16.097952 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:16.097956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:16.098126 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:16.098353 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:16.098455 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.099678 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.101171 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.101979 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.102192 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.102360 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:16.103172 (+ 812us) server_negotiation.cc:299] Negotiation successful
0927 10:53:16.103286 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":102,"threads_started":1}
I20260927 10:53:16.104624 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.096597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35298 (local address 127.14.68.2:32981)
0927 10:53:16.097625 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:16.097638 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:16.097662 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:53:16.097744 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:16.097749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:16.097956 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:16.098143 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:16.098629 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.099809 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.101236 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.101979 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.103573 (+ 1594us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.103784 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:16.104395 (+ 611us) server_negotiation.cc:299] Negotiation successful
0927 10:53:16.104492 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":914,"thread_start_us":113,"threads_started":1}
I20260927 10:53:16.105269 15860 tablet_service.cc:1467] Processing CreateTablet for tablet 9d2c5596a7f449608fe8b45c71fb138b (DEFAULT_TABLE table=test-table [id=75e6efa1ab1241e3ac0a774706653311]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:16.106385 15860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d2c5596a7f449608fe8b45c71fb138b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:16.106123 16140 tablet_service.cc:1467] Processing CreateTablet for tablet 9d2c5596a7f449608fe8b45c71fb138b (DEFAULT_TABLE table=test-table [id=75e6efa1ab1241e3ac0a774706653311]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:16.106923 16140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d2c5596a7f449608fe8b45c71fb138b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:16.107079 16000 tablet_service.cc:1467] Processing CreateTablet for tablet 9d2c5596a7f449608fe8b45c71fb138b (DEFAULT_TABLE table=test-table [id=75e6efa1ab1241e3ac0a774706653311]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:16.107993 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d2c5596a7f449608fe8b45c71fb138b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:16.113391 16229 tablet_bootstrap.cc:492] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83: Bootstrap starting.
I20260927 10:53:16.115190 16231 tablet_bootstrap.cc:492] T 9d2c5596a7f449608fe8b45c71fb138b P 21a6a8d06f2648a09dd95aef8b9f5968: Bootstrap starting.
I20260927 10:53:16.115513 16229 tablet_bootstrap.cc:654] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:16.115788 16230 tablet_bootstrap.cc:492] T 9d2c5596a7f449608fe8b45c71fb138b P fa95d82fe8da48a7a5802d9a197cb479: Bootstrap starting.
I20260927 10:53:16.116354 16229 log.cc:824] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:16.117290 16231 tablet_bootstrap.cc:654] T 9d2c5596a7f449608fe8b45c71fb138b P 21a6a8d06f2648a09dd95aef8b9f5968: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:16.117787 16230 tablet_bootstrap.cc:654] T 9d2c5596a7f449608fe8b45c71fb138b P fa95d82fe8da48a7a5802d9a197cb479: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:16.118106 16231 log.cc:824] T 9d2c5596a7f449608fe8b45c71fb138b P 21a6a8d06f2648a09dd95aef8b9f5968: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:16.118492 16229 tablet_bootstrap.cc:492] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83: No bootstrap required, opened a new log
I20260927 10:53:16.118598 16230 log.cc:824] T 9d2c5596a7f449608fe8b45c71fb138b P fa95d82fe8da48a7a5802d9a197cb479: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:16.118690 16229 ts_tablet_manager.cc:1397] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 10:53:16.120097 16231 tablet_bootstrap.cc:492] T 9d2c5596a7f449608fe8b45c71fb138b P 21a6a8d06f2648a09dd95aef8b9f5968: No bootstrap required, opened a new log
I20260927 10:53:16.120332 16231 ts_tablet_manager.cc:1397] T 9d2c5596a7f449608fe8b45c71fb138b P 21a6a8d06f2648a09dd95aef8b9f5968: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:53:16.120527 16230 tablet_bootstrap.cc:492] T 9d2c5596a7f449608fe8b45c71fb138b P fa95d82fe8da48a7a5802d9a197cb479: No bootstrap required, opened a new log
I20260927 10:53:16.120689 16230 ts_tablet_manager.cc:1397] T 9d2c5596a7f449608fe8b45c71fb138b P fa95d82fe8da48a7a5802d9a197cb479: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:53:16.122278 16229 raft_consensus.cc:348] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } } peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } }
I20260927 10:53:16.122510 16229 raft_consensus.cc:374] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:16.122570 16229 raft_consensus.cc:729] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df8519da8b747b48e64cbee84667d83, State: Initialized, Role: FOLLOWER
I20260927 10:53:16.122994 16229 consensus_queue.cc:260] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } } peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } }
I20260927 10:53:16.123937 16229 ts_tablet_manager.cc:1428] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260927 10:53:16.124217 16230 raft_consensus.cc:348] T 9d2c5596a7f449608fe8b45c71fb138b P fa95d82fe8da48a7a5802d9a197cb479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } } peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } }
I20260927 10:53:16.124217 16231 raft_consensus.cc:348] T 9d2c5596a7f449608fe8b45c71fb138b P 21a6a8d06f2648a09dd95aef8b9f5968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } } peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } }
I20260927 10:53:16.124394 16206 heartbeater.cc:503] Master 127.14.68.62:43327 was elected leader, sending a full tablet report...
I20260927 10:53:16.124450 16231 raft_consensus.cc:374] T 9d2c5596a7f449608fe8b45c71fb138b P 21a6a8d06f2648a09dd95aef8b9f5968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:16.124450 16230 raft_consensus.cc:374] T 9d2c5596a7f449608fe8b45c71fb138b P fa95d82fe8da48a7a5802d9a197cb479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:16.124498 16230 raft_consensus.cc:729] T 9d2c5596a7f449608fe8b45c71fb138b P fa95d82fe8da48a7a5802d9a197cb479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa95d82fe8da48a7a5802d9a197cb479, State: Initialized, Role: FOLLOWER
I20260927 10:53:16.124507 16231 raft_consensus.cc:729] T 9d2c5596a7f449608fe8b45c71fb138b P 21a6a8d06f2648a09dd95aef8b9f5968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21a6a8d06f2648a09dd95aef8b9f5968, State: Initialized, Role: FOLLOWER
I20260927 10:53:16.124936 16231 consensus_queue.cc:260] T 9d2c5596a7f449608fe8b45c71fb138b P 21a6a8d06f2648a09dd95aef8b9f5968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } } peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } }
I20260927 10:53:16.124936 16230 consensus_queue.cc:260] T 9d2c5596a7f449608fe8b45c71fb138b P fa95d82fe8da48a7a5802d9a197cb479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } } peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } }
I20260927 10:53:16.125555 15926 heartbeater.cc:503] Master 127.14.68.62:43327 was elected leader, sending a full tablet report...
I20260927 10:53:16.125840 16231 ts_tablet_manager.cc:1428] T 9d2c5596a7f449608fe8b45c71fb138b P 21a6a8d06f2648a09dd95aef8b9f5968: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:53:16.125869 16066 heartbeater.cc:503] Master 127.14.68.62:43327 was elected leader, sending a full tablet report...
I20260927 10:53:16.127957 16230 ts_tablet_manager.cc:1428] T 9d2c5596a7f449608fe8b45c71fb138b P fa95d82fe8da48a7a5802d9a197cb479: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 10:53:16.258471 16235 raft_consensus.cc:482] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:16.258718 16235 raft_consensus.cc:504] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } } peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } }
I20260927 10:53:16.260047 16235 leader_election.cc:288] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21a6a8d06f2648a09dd95aef8b9f5968 (127.14.68.2:32981), fa95d82fe8da48a7a5802d9a197cb479 (127.14.68.1:37477)
I20260927 10:53:16.263669 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.260482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:43603 (local address 127.14.68.2:32981)
0927 10:53:16.260625 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:16.260634 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:16.260648 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:53:16.260692 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:16.260694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:16.260732 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:16.260821 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:16.261141 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.261645 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.262688 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.263071 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.263198 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.263272 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:16.263475 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 10:53:16.263548 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 10:53:16.263676 16099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.260267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:32981 (local address 127.14.68.3:43603)
0927 10:53:16.260410 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:16.260428 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:16.260522 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:16.260834 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:16.260837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:16.260901 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:16.261018 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.261023 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.261767 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.261773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.262546 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.262554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.263196 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.263200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.263354 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.263372 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 10:53:16.263460 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 10:53:16.263534 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":44,"spinlock_wait_cycles":9984}
I20260927 10:53:16.264386 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.260443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:39401 (local address 127.14.68.1:37477)
0927 10:53:16.260564 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:16.260568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:16.260838 (+ 270us) server_negotiation.cc:407] Connection header received
0927 10:53:16.261058 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:16.261061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:16.261105 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:16.261191 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:16.261851 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.262360 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.263357 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.263833 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.263939 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.263993 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:16.264233 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 10:53:16.264299 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:53:16.264402 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.260359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:37477 (local address 127.14.68.3:39401)
0927 10:53:16.260731 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:16.260741 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:16.260917 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:16.261211 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:16.261214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:16.261230 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:16.261710 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.261717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.262470 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.262476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.263234 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.263242 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.263950 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.263954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.264086 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.264120 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 10:53:16.264200 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 10:53:16.264264 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":1,"spinlock_wait_cycles":29568,"thread_start_us":124,"threads_started":1}
I20260927 10:53:16.264410 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2c5596a7f449608fe8b45c71fb138b" candidate_uuid: "1df8519da8b747b48e64cbee84667d83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" is_pre_election: true
I20260927 10:53:16.264726 16020 raft_consensus.cc:2454] T 9d2c5596a7f449608fe8b45c71fb138b P 21a6a8d06f2648a09dd95aef8b9f5968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1df8519da8b747b48e64cbee84667d83 in term 0.
I20260927 10:53:16.265024 15880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2c5596a7f449608fe8b45c71fb138b" candidate_uuid: "1df8519da8b747b48e64cbee84667d83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa95d82fe8da48a7a5802d9a197cb479" is_pre_election: true
I20260927 10:53:16.265261 16091 leader_election.cc:302] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df8519da8b747b48e64cbee84667d83, 21a6a8d06f2648a09dd95aef8b9f5968; no voters:
I20260927 10:53:16.265342 15880 raft_consensus.cc:2454] T 9d2c5596a7f449608fe8b45c71fb138b P fa95d82fe8da48a7a5802d9a197cb479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1df8519da8b747b48e64cbee84667d83 in term 0.
I20260927 10:53:16.265530 16235 raft_consensus.cc:2781] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:16.265628 16235 raft_consensus.cc:482] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:16.265710 16235 raft_consensus.cc:3037] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:16.266759 16235 raft_consensus.cc:504] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } } peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } }
I20260927 10:53:16.267099 16235 leader_election.cc:288] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83 [CANDIDATE]: Term 1 election: Requested vote from peers 21a6a8d06f2648a09dd95aef8b9f5968 (127.14.68.2:32981), fa95d82fe8da48a7a5802d9a197cb479 (127.14.68.1:37477)
I20260927 10:53:16.267441 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2c5596a7f449608fe8b45c71fb138b" candidate_uuid: "1df8519da8b747b48e64cbee84667d83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21a6a8d06f2648a09dd95aef8b9f5968"
I20260927 10:53:16.267468 15880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2c5596a7f449608fe8b45c71fb138b" candidate_uuid: "1df8519da8b747b48e64cbee84667d83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa95d82fe8da48a7a5802d9a197cb479"
I20260927 10:53:16.267550 16020 raft_consensus.cc:3037] T 9d2c5596a7f449608fe8b45c71fb138b P 21a6a8d06f2648a09dd95aef8b9f5968 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:16.267613 15880 raft_consensus.cc:3037] T 9d2c5596a7f449608fe8b45c71fb138b P fa95d82fe8da48a7a5802d9a197cb479 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:16.268639 15880 raft_consensus.cc:2454] T 9d2c5596a7f449608fe8b45c71fb138b P fa95d82fe8da48a7a5802d9a197cb479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df8519da8b747b48e64cbee84667d83 in term 1.
I20260927 10:53:16.268927 16093 leader_election.cc:302] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df8519da8b747b48e64cbee84667d83, fa95d82fe8da48a7a5802d9a197cb479; no voters:
I20260927 10:53:16.269109 16235 raft_consensus.cc:2781] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:16.269254 16020 raft_consensus.cc:2454] T 9d2c5596a7f449608fe8b45c71fb138b P 21a6a8d06f2648a09dd95aef8b9f5968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df8519da8b747b48e64cbee84667d83 in term 1.
I20260927 10:53:16.269325 16235 raft_consensus.cc:686] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83 [term 1 LEADER]: Becoming Leader. State: Replica: 1df8519da8b747b48e64cbee84667d83, State: Running, Role: LEADER
I20260927 10:53:16.269680 16235 consensus_queue.cc:237] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } } peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } }
I20260927 10:53:16.272909 15722 catalog_manager.cc:5495] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1df8519da8b747b48e64cbee84667d83 (127.14.68.3). New cstate: current_term: 1 leader_uuid: "1df8519da8b747b48e64cbee84667d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } health_report { overall_health: UNKNOWN } } }
I20260927 10:53:16.293341 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.289528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53230 (local address 127.14.68.3:45303)
0927 10:53:16.289684 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:16.289688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:16.289703 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:53:16.289889 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:16.289895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:16.289955 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:16.290057 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:16.290339 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.290852 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.291746 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.291967 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.292076 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.292409 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:16.292911 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:16.293033 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:16.293131 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 10:53:16.293190 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 10:53:16.296231 16207 tablet.cc:2369] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:53:16.298827 15880 raft_consensus.cc:1261] T 9d2c5596a7f449608fe8b45c71fb138b P fa95d82fe8da48a7a5802d9a197cb479 [term 1 FOLLOWER]: Refusing update from remote peer 1df8519da8b747b48e64cbee84667d83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:16.298836 16020 raft_consensus.cc:1261] T 9d2c5596a7f449608fe8b45c71fb138b P 21a6a8d06f2648a09dd95aef8b9f5968 [term 1 FOLLOWER]: Refusing update from remote peer 1df8519da8b747b48e64cbee84667d83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:16.299521 16239 consensus_queue.cc:1035] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:16.299754 16235 consensus_queue.cc:1035] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:16.307212 16244 mvcc.cc:205] Tried to move back new op lower bound from 7333914199233482752 to 7333914199123472384. Current Snapshot: MvccSnapshot[applied={T|T < 7333914199233482752}]
I20260927 10:53:16.318269 15723 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:16.318442 15723 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:16.320909 15723 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541: Sending DeleteTablet for 3 replicas of tablet 9d2c5596a7f449608fe8b45c71fb138b
I20260927 10:53:16.321628 15860 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d2c5596a7f449608fe8b45c71fb138b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:16 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:52246
I20260927 10:53:16.321810 16000 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d2c5596a7f449608fe8b45c71fb138b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:16 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:35298
I20260927 10:53:16.321830 16140 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d2c5596a7f449608fe8b45c71fb138b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:16 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:53214
I20260927 10:53:16.322232 16252 tablet_replica.cc:330] T 9d2c5596a7f449608fe8b45c71fb138b P fa95d82fe8da48a7a5802d9a197cb479: stopping tablet replica
I20260927 10:53:16.322543 16252 raft_consensus.cc:2229] T 9d2c5596a7f449608fe8b45c71fb138b P fa95d82fe8da48a7a5802d9a197cb479 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:16.322853 16253 tablet_replica.cc:330] T 9d2c5596a7f449608fe8b45c71fb138b P 21a6a8d06f2648a09dd95aef8b9f5968: stopping tablet replica
I20260927 10:53:16.322853 16252 raft_consensus.cc:2258] T 9d2c5596a7f449608fe8b45c71fb138b P fa95d82fe8da48a7a5802d9a197cb479 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:16.322840 15723 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 10:53:16.323098 16253 raft_consensus.cc:2229] T 9d2c5596a7f449608fe8b45c71fb138b P 21a6a8d06f2648a09dd95aef8b9f5968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:16.323355 16253 raft_consensus.cc:2258] T 9d2c5596a7f449608fe8b45c71fb138b P 21a6a8d06f2648a09dd95aef8b9f5968 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 10:53:16.323581 15723 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:16.324694 16252 ts_tablet_manager.cc:1905] T 9d2c5596a7f449608fe8b45c71fb138b P fa95d82fe8da48a7a5802d9a197cb479: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:16.324723 16253 ts_tablet_manager.cc:1905] T 9d2c5596a7f449608fe8b45c71fb138b P 21a6a8d06f2648a09dd95aef8b9f5968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:16.327418 16253 ts_tablet_manager.cc:1918] T 9d2c5596a7f449608fe8b45c71fb138b P 21a6a8d06f2648a09dd95aef8b9f5968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:16.327512 16253 log.cc:1196] T 9d2c5596a7f449608fe8b45c71fb138b P 21a6a8d06f2648a09dd95aef8b9f5968: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/9d2c5596a7f449608fe8b45c71fb138b
I20260927 10:53:16.327889 16253 ts_tablet_manager.cc:1939] T 9d2c5596a7f449608fe8b45c71fb138b P 21a6a8d06f2648a09dd95aef8b9f5968: Deleting consensus metadata
I20260927 10:53:16.328161 16000 tablet_service.cc:1467] Processing CreateTablet for tablet df37cc5246bc480e9b255a4507706b2e (DEFAULT_TABLE table=test-table [id=bdce35b41df8445cae1aa138a54abec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:16.328197 15860 tablet_service.cc:1467] Processing CreateTablet for tablet df37cc5246bc480e9b255a4507706b2e (DEFAULT_TABLE table=test-table [id=bdce35b41df8445cae1aa138a54abec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:16.328382 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df37cc5246bc480e9b255a4507706b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:16.328420 15860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df37cc5246bc480e9b255a4507706b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:16.328990 15707 catalog_manager.cc:4841] TS 21a6a8d06f2648a09dd95aef8b9f5968 (127.14.68.2:32981): tablet 9d2c5596a7f449608fe8b45c71fb138b (table test-table [id=75e6efa1ab1241e3ac0a774706653311]) successfully deleted
I20260927 10:53:16.329274 16254 tablet_replica.cc:330] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83: stopping tablet replica
I20260927 10:53:16.329527 16254 raft_consensus.cc:2229] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:53:16.329895 16254 raft_consensus.cc:2258] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:16.330377 16231 tablet_bootstrap.cc:492] T df37cc5246bc480e9b255a4507706b2e P 21a6a8d06f2648a09dd95aef8b9f5968: Bootstrap starting.
I20260927 10:53:16.330791 16230 tablet_bootstrap.cc:492] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479: Bootstrap starting.
I20260927 10:53:16.331300 16231 tablet_bootstrap.cc:654] T df37cc5246bc480e9b255a4507706b2e P 21a6a8d06f2648a09dd95aef8b9f5968: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:16.331748 16230 tablet_bootstrap.cc:654] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:16.331991 16254 ts_tablet_manager.cc:1905] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:16.332895 16230 tablet_bootstrap.cc:492] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479: No bootstrap required, opened a new log
I20260927 10:53:16.332971 16230 ts_tablet_manager.cc:1397] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 10:53:16.333397 16230 raft_consensus.cc:348] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } }
I20260927 10:53:16.333513 16230 raft_consensus.cc:374] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:16.333549 16230 raft_consensus.cc:729] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa95d82fe8da48a7a5802d9a197cb479, State: Initialized, Role: FOLLOWER
I20260927 10:53:16.333676 16230 consensus_queue.cc:260] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } }
I20260927 10:53:16.333837 16140 tablet_service.cc:1467] Processing CreateTablet for tablet df37cc5246bc480e9b255a4507706b2e (DEFAULT_TABLE table=test-table [id=bdce35b41df8445cae1aa138a54abec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:16.333914 16230 ts_tablet_manager.cc:1428] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 10:53:16.334071 16140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df37cc5246bc480e9b255a4507706b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:16.334440 16254 ts_tablet_manager.cc:1918] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:16.334527 16254 log.cc:1196] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/9d2c5596a7f449608fe8b45c71fb138b
I20260927 10:53:16.334805 16254 ts_tablet_manager.cc:1939] T 9d2c5596a7f449608fe8b45c71fb138b P 1df8519da8b747b48e64cbee84667d83: Deleting consensus metadata
I20260927 10:53:16.336344 16229 tablet_bootstrap.cc:492] T df37cc5246bc480e9b255a4507706b2e P 1df8519da8b747b48e64cbee84667d83: Bootstrap starting.
I20260927 10:53:16.336493 15707 catalog_manager.cc:4841] TS 1df8519da8b747b48e64cbee84667d83 (127.14.68.3:45303): tablet 9d2c5596a7f449608fe8b45c71fb138b (table test-table [id=75e6efa1ab1241e3ac0a774706653311]) successfully deleted
I20260927 10:53:16.337324 16229 tablet_bootstrap.cc:654] T df37cc5246bc480e9b255a4507706b2e P 1df8519da8b747b48e64cbee84667d83: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:16.338768 16229 tablet_bootstrap.cc:492] T df37cc5246bc480e9b255a4507706b2e P 1df8519da8b747b48e64cbee84667d83: No bootstrap required, opened a new log
I20260927 10:53:16.338852 16229 ts_tablet_manager.cc:1397] T df37cc5246bc480e9b255a4507706b2e P 1df8519da8b747b48e64cbee84667d83: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:53:16.339313 16229 raft_consensus.cc:348] T df37cc5246bc480e9b255a4507706b2e P 1df8519da8b747b48e64cbee84667d83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } }
I20260927 10:53:16.339461 16229 raft_consensus.cc:374] T df37cc5246bc480e9b255a4507706b2e P 1df8519da8b747b48e64cbee84667d83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:16.339499 16229 raft_consensus.cc:729] T df37cc5246bc480e9b255a4507706b2e P 1df8519da8b747b48e64cbee84667d83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df8519da8b747b48e64cbee84667d83, State: Initialized, Role: FOLLOWER
I20260927 10:53:16.339637 16229 consensus_queue.cc:260] T df37cc5246bc480e9b255a4507706b2e P 1df8519da8b747b48e64cbee84667d83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } }
I20260927 10:53:16.339972 16229 ts_tablet_manager.cc:1428] T df37cc5246bc480e9b255a4507706b2e P 1df8519da8b747b48e64cbee84667d83: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:53:16.340022 16252 ts_tablet_manager.cc:1918] T 9d2c5596a7f449608fe8b45c71fb138b P fa95d82fe8da48a7a5802d9a197cb479: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:16.340112 16252 log.cc:1196] T 9d2c5596a7f449608fe8b45c71fb138b P fa95d82fe8da48a7a5802d9a197cb479: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-0/wal/wals/9d2c5596a7f449608fe8b45c71fb138b
I20260927 10:53:16.340453 16252 ts_tablet_manager.cc:1939] T 9d2c5596a7f449608fe8b45c71fb138b P fa95d82fe8da48a7a5802d9a197cb479: Deleting consensus metadata
I20260927 10:53:16.341800 15709 catalog_manager.cc:4841] TS fa95d82fe8da48a7a5802d9a197cb479 (127.14.68.1:37477): tablet 9d2c5596a7f449608fe8b45c71fb138b (table test-table [id=75e6efa1ab1241e3ac0a774706653311]) successfully deleted
I20260927 10:53:16.342407 16231 tablet_bootstrap.cc:492] T df37cc5246bc480e9b255a4507706b2e P 21a6a8d06f2648a09dd95aef8b9f5968: No bootstrap required, opened a new log
I20260927 10:53:16.342526 16231 ts_tablet_manager.cc:1397] T df37cc5246bc480e9b255a4507706b2e P 21a6a8d06f2648a09dd95aef8b9f5968: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260927 10:53:16.342968 16231 raft_consensus.cc:348] T df37cc5246bc480e9b255a4507706b2e P 21a6a8d06f2648a09dd95aef8b9f5968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } }
I20260927 10:53:16.343089 16231 raft_consensus.cc:374] T df37cc5246bc480e9b255a4507706b2e P 21a6a8d06f2648a09dd95aef8b9f5968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:16.343144 16231 raft_consensus.cc:729] T df37cc5246bc480e9b255a4507706b2e P 21a6a8d06f2648a09dd95aef8b9f5968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21a6a8d06f2648a09dd95aef8b9f5968, State: Initialized, Role: FOLLOWER
I20260927 10:53:16.343298 16231 consensus_queue.cc:260] T df37cc5246bc480e9b255a4507706b2e P 21a6a8d06f2648a09dd95aef8b9f5968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } }
I20260927 10:53:16.343551 16231 ts_tablet_manager.cc:1428] T df37cc5246bc480e9b255a4507706b2e P 21a6a8d06f2648a09dd95aef8b9f5968: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260927 10:53:16.364758 15927 tablet.cc:2369] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:53:16.450812 16236 raft_consensus.cc:482] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:16.450974 16236 raft_consensus.cc:504] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } }
I20260927 10:53:16.452261 16236 leader_election.cc:288] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21a6a8d06f2648a09dd95aef8b9f5968 (127.14.68.2:32981), 1df8519da8b747b48e64cbee84667d83 (127.14.68.3:45303)
I20260927 10:53:16.457258 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.453095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:52501 (local address 127.14.68.3:45303)
0927 10:53:16.453210 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:16.453214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:16.453452 (+ 238us) server_negotiation.cc:407] Connection header received
0927 10:53:16.453557 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:16.453560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:16.453605 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:16.453692 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:16.453994 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.454482 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.456017 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.456617 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.456730 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.456800 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:16.457047 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 10:53:16.457125 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:53:16.457294 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.452997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:45303 (local address 127.14.68.1:52501)
0927 10:53:16.453365 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:16.453379 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:16.453468 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:16.453721 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:16.453726 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:16.453747 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:16.453870 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.453884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.454622 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.454625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.455888 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.455898 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.456759 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.456763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.456873 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.456909 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 10:53:16.457030 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 10:53:16.457118 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":36,"thread_start_us":105,"threads_started":1}
I20260927 10:53:16.457379 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.452690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:46765 (local address 127.14.68.2:32981)
0927 10:53:16.452833 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:16.452838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:16.452983 (+ 145us) server_negotiation.cc:407] Connection header received
0927 10:53:16.453277 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:16.453280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:16.453334 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:16.453386 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:16.453867 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.454650 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.456167 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.456800 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.456899 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.456954 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:16.457163 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 10:53:16.457247 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:53:16.457815 16160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df37cc5246bc480e9b255a4507706b2e" candidate_uuid: "fa95d82fe8da48a7a5802d9a197cb479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df8519da8b747b48e64cbee84667d83" is_pre_election: true
I20260927 10:53:16.457899 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.452618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:32981 (local address 127.14.68.1:46765)
0927 10:53:16.452896 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:16.452913 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:16.453125 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:16.453539 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:16.453545 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:16.453563 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:16.453696 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.453705 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.454781 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.454785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.456038 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.456050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.456901 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.456905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.457015 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.457042 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:53:16.457690 (+ 648us) client_negotiation.cc:240] Negotiation successful
0927 10:53:16.457762 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":1,"thread_start_us":96,"threads_started":1}
I20260927 10:53:16.458057 16160 raft_consensus.cc:2454] T df37cc5246bc480e9b255a4507706b2e P 1df8519da8b747b48e64cbee84667d83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa95d82fe8da48a7a5802d9a197cb479 in term 0.
I20260927 10:53:16.458386 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df37cc5246bc480e9b255a4507706b2e" candidate_uuid: "fa95d82fe8da48a7a5802d9a197cb479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" is_pre_election: true
I20260927 10:53:16.458474 15811 leader_election.cc:302] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df8519da8b747b48e64cbee84667d83, fa95d82fe8da48a7a5802d9a197cb479; no voters:
I20260927 10:53:16.458536 16020 raft_consensus.cc:2454] T df37cc5246bc480e9b255a4507706b2e P 21a6a8d06f2648a09dd95aef8b9f5968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa95d82fe8da48a7a5802d9a197cb479 in term 0.
I20260927 10:53:16.458760 16236 raft_consensus.cc:2781] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:16.458818 16236 raft_consensus.cc:482] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:16.458865 16236 raft_consensus.cc:3037] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:16.459874 16236 raft_consensus.cc:504] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } }
I20260927 10:53:16.460251 16236 leader_election.cc:288] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479 [CANDIDATE]: Term 1 election: Requested vote from peers 21a6a8d06f2648a09dd95aef8b9f5968 (127.14.68.2:32981), 1df8519da8b747b48e64cbee84667d83 (127.14.68.3:45303)
I20260927 10:53:16.460572 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df37cc5246bc480e9b255a4507706b2e" candidate_uuid: "fa95d82fe8da48a7a5802d9a197cb479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21a6a8d06f2648a09dd95aef8b9f5968"
I20260927 10:53:16.460592 16160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df37cc5246bc480e9b255a4507706b2e" candidate_uuid: "fa95d82fe8da48a7a5802d9a197cb479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df8519da8b747b48e64cbee84667d83"
I20260927 10:53:16.460700 16160 raft_consensus.cc:3037] T df37cc5246bc480e9b255a4507706b2e P 1df8519da8b747b48e64cbee84667d83 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:16.460698 16020 raft_consensus.cc:3037] T df37cc5246bc480e9b255a4507706b2e P 21a6a8d06f2648a09dd95aef8b9f5968 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:16.461570 16020 raft_consensus.cc:2454] T df37cc5246bc480e9b255a4507706b2e P 21a6a8d06f2648a09dd95aef8b9f5968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa95d82fe8da48a7a5802d9a197cb479 in term 1.
I20260927 10:53:16.461824 16160 raft_consensus.cc:2454] T df37cc5246bc480e9b255a4507706b2e P 1df8519da8b747b48e64cbee84667d83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa95d82fe8da48a7a5802d9a197cb479 in term 1.
I20260927 10:53:16.461828 15811 leader_election.cc:302] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21a6a8d06f2648a09dd95aef8b9f5968, fa95d82fe8da48a7a5802d9a197cb479; no voters:
I20260927 10:53:16.461997 16236 raft_consensus.cc:2781] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:16.462203 16236 raft_consensus.cc:686] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479 [term 1 LEADER]: Becoming Leader. State: Replica: fa95d82fe8da48a7a5802d9a197cb479, State: Running, Role: LEADER
I20260927 10:53:16.462450 16236 consensus_queue.cc:237] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } }
I20260927 10:53:16.464632 15722 catalog_manager.cc:5495] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa95d82fe8da48a7a5802d9a197cb479 (127.14.68.1). New cstate: current_term: 1 leader_uuid: "fa95d82fe8da48a7a5802d9a197cb479" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } health_report { overall_health: UNKNOWN } } }
Sep 27 10:53:16 dist-test-slave-tz51 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506395, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:16.513087 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.505210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:43327 (local address 127.0.0.1:41000)
0927 10:53:16.505474 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:16.505487 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:16.505580 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:16.505795 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:16.505797 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:16.506229 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:16.506317 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.506323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.507079 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.507081 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.507651 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.507660 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.508159 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.508162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.508225 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.508750 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:16.508762 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:16.510270 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:16.511654 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:16.511662 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:16.511667 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:16.511909 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:16.512159 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:16.512161 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:16.512162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:16.512235 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:16.512611 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:16.512616 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:16.512707 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 10:53:16.512790 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 10:53:16.512888 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":82,"threads_started":1}
I20260927 10:53:16.513099 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.505329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41000 (local address 127.14.68.62:43327)
0927 10:53:16.505466 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:16.505471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:16.505553 (+ 82us) server_negotiation.cc:407] Connection header received
0927 10:53:16.505655 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:16.505657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:16.505695 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:16.505762 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:16.506424 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.506931 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.507848 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.508051 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.508157 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.510394 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:16.510397 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:16.510399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:16.510419 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:16.511565 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:16.511990 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:16.511992 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:16.511993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:16.512069 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:16.512340 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:16.512343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:16.512344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:16.512519 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:16.512606 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:16.512848 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 10:53:16.512937 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:53:16.516920 15722 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:41000:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 10:53:16.517566 15722 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:16.522392 15860 tablet_service.cc:1467] Processing CreateTablet for tablet 8731d83873ae46a9ac29073a3d3e81bb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b53c50ac7234548b7512426aa2b4cf5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:53:16.522488 16000 tablet_service.cc:1467] Processing CreateTablet for tablet 8731d83873ae46a9ac29073a3d3e81bb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b53c50ac7234548b7512426aa2b4cf5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:53:16.522541 16140 tablet_service.cc:1467] Processing CreateTablet for tablet 8731d83873ae46a9ac29073a3d3e81bb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b53c50ac7234548b7512426aa2b4cf5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:53:16.522789 15860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8731d83873ae46a9ac29073a3d3e81bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:16.522789 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8731d83873ae46a9ac29073a3d3e81bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:16.522841 16140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8731d83873ae46a9ac29073a3d3e81bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:16.525175 16229 tablet_bootstrap.cc:492] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83: Bootstrap starting.
I20260927 10:53:16.525593 16230 tablet_bootstrap.cc:492] T 8731d83873ae46a9ac29073a3d3e81bb P fa95d82fe8da48a7a5802d9a197cb479: Bootstrap starting.
I20260927 10:53:16.526090 16229 tablet_bootstrap.cc:654] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:16.526578 16230 tablet_bootstrap.cc:654] T 8731d83873ae46a9ac29073a3d3e81bb P fa95d82fe8da48a7a5802d9a197cb479: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:16.527244 16231 tablet_bootstrap.cc:492] T 8731d83873ae46a9ac29073a3d3e81bb P 21a6a8d06f2648a09dd95aef8b9f5968: Bootstrap starting.
I20260927 10:53:16.527259 16229 tablet_bootstrap.cc:492] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83: No bootstrap required, opened a new log
I20260927 10:53:16.527331 16229 ts_tablet_manager.cc:1397] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:53:16.527745 16229 raft_consensus.cc:348] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } }
I20260927 10:53:16.527868 16229 raft_consensus.cc:374] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:16.527901 16229 raft_consensus.cc:729] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df8519da8b747b48e64cbee84667d83, State: Initialized, Role: FOLLOWER
I20260927 10:53:16.527916 16230 tablet_bootstrap.cc:492] T 8731d83873ae46a9ac29073a3d3e81bb P fa95d82fe8da48a7a5802d9a197cb479: No bootstrap required, opened a new log
I20260927 10:53:16.527995 16230 ts_tablet_manager.cc:1397] T 8731d83873ae46a9ac29073a3d3e81bb P fa95d82fe8da48a7a5802d9a197cb479: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 10:53:16.528018 16229 consensus_queue.cc:260] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } }
I20260927 10:53:16.528169 16231 tablet_bootstrap.cc:654] T 8731d83873ae46a9ac29073a3d3e81bb P 21a6a8d06f2648a09dd95aef8b9f5968: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:16.528347 16229 ts_tablet_manager.cc:1428] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:53:16.528435 16230 raft_consensus.cc:348] T 8731d83873ae46a9ac29073a3d3e81bb P fa95d82fe8da48a7a5802d9a197cb479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } }
I20260927 10:53:16.528563 16230 raft_consensus.cc:374] T 8731d83873ae46a9ac29073a3d3e81bb P fa95d82fe8da48a7a5802d9a197cb479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:16.528594 16230 raft_consensus.cc:729] T 8731d83873ae46a9ac29073a3d3e81bb P fa95d82fe8da48a7a5802d9a197cb479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa95d82fe8da48a7a5802d9a197cb479, State: Initialized, Role: FOLLOWER
I20260927 10:53:16.528620 16249 tablet_replica.cc:439] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } } }
I20260927 10:53:16.528710 16230 consensus_queue.cc:260] T 8731d83873ae46a9ac29073a3d3e81bb P fa95d82fe8da48a7a5802d9a197cb479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } }
I20260927 10:53:16.528751 16249 tablet_replica.cc:442] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:53:16.528940 16236 tablet_replica.cc:439] T 8731d83873ae46a9ac29073a3d3e81bb P fa95d82fe8da48a7a5802d9a197cb479: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } } }
I20260927 10:53:16.529070 16236 tablet_replica.cc:442] T 8731d83873ae46a9ac29073a3d3e81bb P fa95d82fe8da48a7a5802d9a197cb479: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:53:16.529913 16231 tablet_bootstrap.cc:492] T 8731d83873ae46a9ac29073a3d3e81bb P 21a6a8d06f2648a09dd95aef8b9f5968: No bootstrap required, opened a new log
I20260927 10:53:16.530009 16231 ts_tablet_manager.cc:1397] T 8731d83873ae46a9ac29073a3d3e81bb P 21a6a8d06f2648a09dd95aef8b9f5968: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:53:16.530445 16231 raft_consensus.cc:348] T 8731d83873ae46a9ac29073a3d3e81bb P 21a6a8d06f2648a09dd95aef8b9f5968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } }
I20260927 10:53:16.530594 16231 raft_consensus.cc:374] T 8731d83873ae46a9ac29073a3d3e81bb P 21a6a8d06f2648a09dd95aef8b9f5968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:16.530644 16231 raft_consensus.cc:729] T 8731d83873ae46a9ac29073a3d3e81bb P 21a6a8d06f2648a09dd95aef8b9f5968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21a6a8d06f2648a09dd95aef8b9f5968, State: Initialized, Role: FOLLOWER
I20260927 10:53:16.530769 16231 consensus_queue.cc:260] T 8731d83873ae46a9ac29073a3d3e81bb P 21a6a8d06f2648a09dd95aef8b9f5968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } }
I20260927 10:53:16.530997 16237 tablet_replica.cc:439] T 8731d83873ae46a9ac29073a3d3e81bb P 21a6a8d06f2648a09dd95aef8b9f5968: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } } }
I20260927 10:53:16.531122 16237 tablet_replica.cc:442] T 8731d83873ae46a9ac29073a3d3e81bb P 21a6a8d06f2648a09dd95aef8b9f5968: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:53:16.531373 16231 ts_tablet_manager.cc:1428] T 8731d83873ae46a9ac29073a3d3e81bb P 21a6a8d06f2648a09dd95aef8b9f5968: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:53:16.528956 16230 ts_tablet_manager.cc:1428] T 8731d83873ae46a9ac29073a3d3e81bb P fa95d82fe8da48a7a5802d9a197cb479: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260927 10:53:16.586575 16067 tablet.cc:2369] T 8731d83873ae46a9ac29073a3d3e81bb P 21a6a8d06f2648a09dd95aef8b9f5968: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:53:16.668048 16249 raft_consensus.cc:482] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:16.668226 16249 raft_consensus.cc:504] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } }
I20260927 10:53:16.668707 16249 leader_election.cc:288] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa95d82fe8da48a7a5802d9a197cb479 (127.14.68.1:37477), 21a6a8d06f2648a09dd95aef8b9f5968 (127.14.68.2:32981)
I20260927 10:53:16.669318 15880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8731d83873ae46a9ac29073a3d3e81bb" candidate_uuid: "1df8519da8b747b48e64cbee84667d83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa95d82fe8da48a7a5802d9a197cb479" is_pre_election: true
I20260927 10:53:16.669318 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8731d83873ae46a9ac29073a3d3e81bb" candidate_uuid: "1df8519da8b747b48e64cbee84667d83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" is_pre_election: true
I20260927 10:53:16.669481 16020 raft_consensus.cc:2454] T 8731d83873ae46a9ac29073a3d3e81bb P 21a6a8d06f2648a09dd95aef8b9f5968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1df8519da8b747b48e64cbee84667d83 in term 0.
I20260927 10:53:16.669483 15880 raft_consensus.cc:2454] T 8731d83873ae46a9ac29073a3d3e81bb P fa95d82fe8da48a7a5802d9a197cb479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1df8519da8b747b48e64cbee84667d83 in term 0.
I20260927 10:53:16.669843 16093 leader_election.cc:302] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df8519da8b747b48e64cbee84667d83, fa95d82fe8da48a7a5802d9a197cb479; no voters:
I20260927 10:53:16.670060 16249 raft_consensus.cc:2781] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:16.670122 16249 raft_consensus.cc:482] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:16.670156 16249 raft_consensus.cc:3037] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:16.671214 16249 raft_consensus.cc:504] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } }
I20260927 10:53:16.671559 16249 leader_election.cc:288] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83 [CANDIDATE]: Term 1 election: Requested vote from peers fa95d82fe8da48a7a5802d9a197cb479 (127.14.68.1:37477), 21a6a8d06f2648a09dd95aef8b9f5968 (127.14.68.2:32981)
I20260927 10:53:16.672008 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8731d83873ae46a9ac29073a3d3e81bb" candidate_uuid: "1df8519da8b747b48e64cbee84667d83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21a6a8d06f2648a09dd95aef8b9f5968"
I20260927 10:53:16.671994 15880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8731d83873ae46a9ac29073a3d3e81bb" candidate_uuid: "1df8519da8b747b48e64cbee84667d83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa95d82fe8da48a7a5802d9a197cb479"
I20260927 10:53:16.672155 16020 raft_consensus.cc:3037] T 8731d83873ae46a9ac29073a3d3e81bb P 21a6a8d06f2648a09dd95aef8b9f5968 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:16.672186 15880 raft_consensus.cc:3037] T 8731d83873ae46a9ac29073a3d3e81bb P fa95d82fe8da48a7a5802d9a197cb479 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:16.673131 16020 raft_consensus.cc:2454] T 8731d83873ae46a9ac29073a3d3e81bb P 21a6a8d06f2648a09dd95aef8b9f5968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df8519da8b747b48e64cbee84667d83 in term 1.
I20260927 10:53:16.673219 15880 raft_consensus.cc:2454] T 8731d83873ae46a9ac29073a3d3e81bb P fa95d82fe8da48a7a5802d9a197cb479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df8519da8b747b48e64cbee84667d83 in term 1.
I20260927 10:53:16.673462 16091 leader_election.cc:302] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df8519da8b747b48e64cbee84667d83, 21a6a8d06f2648a09dd95aef8b9f5968; no voters:
I20260927 10:53:16.673623 16249 raft_consensus.cc:2781] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:16.673741 16249 raft_consensus.cc:686] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83 [term 1 LEADER]: Becoming Leader. State: Replica: 1df8519da8b747b48e64cbee84667d83, State: Running, Role: LEADER
I20260927 10:53:16.673906 16249 consensus_queue.cc:237] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } }
I20260927 10:53:16.674705 16235 tablet_replica.cc:439] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83: TxnStatusTablet state changed. Reason: New leader 1df8519da8b747b48e64cbee84667d83. Latest consensus state: current_term: 1 leader_uuid: "1df8519da8b747b48e64cbee84667d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } } }
I20260927 10:53:16.674801 16235 tablet_replica.cc:442] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:53:16.675025 16272 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 10:53:16.675208 16272 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 10:53:16.675951 15723 catalog_manager.cc:5495] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1df8519da8b747b48e64cbee84667d83 (127.14.68.3). New cstate: current_term: 1 leader_uuid: "1df8519da8b747b48e64cbee84667d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } health_report { overall_health: HEALTHY } } }
I20260927 10:53:16.722337 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.718623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53236 (local address 127.14.68.3:45303)
0927 10:53:16.718779 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:16.718783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:16.718797 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:53:16.718956 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:16.718959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:16.719005 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:16.719073 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:16.719347 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.720066 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.721272 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.721484 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.721565 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.721884 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:16.721955 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:16.722008 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:16.722158 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 10:53:16.722215 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:53:16.722368 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.718529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:45303 (local address 127.0.0.1:53236)
0927 10:53:16.718708 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:16.718724 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:16.718833 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:16.719115 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:16.719118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:16.719128 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:53:16.719242 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.719249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.720177 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.720181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.721154 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.721164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.721594 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.721597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.721698 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.721753 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:53:16.722036 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:53:16.722049 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 10:53:16.722134 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 10:53:16.722220 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 10:53:16.725214 16020 raft_consensus.cc:1261] T 8731d83873ae46a9ac29073a3d3e81bb P 21a6a8d06f2648a09dd95aef8b9f5968 [term 1 FOLLOWER]: Refusing update from remote peer 1df8519da8b747b48e64cbee84667d83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:16.725328 15880 raft_consensus.cc:1261] T 8731d83873ae46a9ac29073a3d3e81bb P fa95d82fe8da48a7a5802d9a197cb479 [term 1 FOLLOWER]: Refusing update from remote peer 1df8519da8b747b48e64cbee84667d83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:16.725694 16235 consensus_queue.cc:1035] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:16.726001 16235 consensus_queue.cc:1035] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:16.727334 16237 tablet_replica.cc:439] T 8731d83873ae46a9ac29073a3d3e81bb P 21a6a8d06f2648a09dd95aef8b9f5968: TxnStatusTablet state changed. Reason: New leader 1df8519da8b747b48e64cbee84667d83. Latest consensus state: current_term: 1 leader_uuid: "1df8519da8b747b48e64cbee84667d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } } }
I20260927 10:53:16.727459 16237 tablet_replica.cc:442] T 8731d83873ae46a9ac29073a3d3e81bb P 21a6a8d06f2648a09dd95aef8b9f5968: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:53:16.727402 16236 tablet_replica.cc:439] T 8731d83873ae46a9ac29073a3d3e81bb P fa95d82fe8da48a7a5802d9a197cb479: TxnStatusTablet state changed. Reason: New leader 1df8519da8b747b48e64cbee84667d83. Latest consensus state: current_term: 1 leader_uuid: "1df8519da8b747b48e64cbee84667d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } } }
I20260927 10:53:16.727505 16236 tablet_replica.cc:442] T 8731d83873ae46a9ac29073a3d3e81bb P fa95d82fe8da48a7a5802d9a197cb479: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:53:16.728513 16249 tablet_replica.cc:439] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1df8519da8b747b48e64cbee84667d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } } }
I20260927 10:53:16.728615 16249 tablet_replica.cc:442] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:53:16.728713 16242 mvcc.cc:205] Tried to move back new op lower bound from 7333914200982159360 to 7333914200777961472. Current Snapshot: MvccSnapshot[applied={T|T < 7333914200982159360}]
I20260927 10:53:16.731017 16239 tablet_replica.cc:439] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1df8519da8b747b48e64cbee84667d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } } }
I20260927 10:53:16.731114 16239 tablet_replica.cc:442] T 8731d83873ae46a9ac29073a3d3e81bb P 1df8519da8b747b48e64cbee84667d83: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:53:16.731563 16243 mvcc.cc:205] Tried to move back new op lower bound from 7333914200982159360 to 7333914200777961472. Current Snapshot: MvccSnapshot[applied={T|T < 7333914200982159360}]
I20260927 10:53:16.731597 16236 tablet_replica.cc:439] T 8731d83873ae46a9ac29073a3d3e81bb P fa95d82fe8da48a7a5802d9a197cb479: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1df8519da8b747b48e64cbee84667d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } } }
I20260927 10:53:16.731726 16236 tablet_replica.cc:442] T 8731d83873ae46a9ac29073a3d3e81bb P fa95d82fe8da48a7a5802d9a197cb479: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:53:16.732939 16237 tablet_replica.cc:439] T 8731d83873ae46a9ac29073a3d3e81bb P 21a6a8d06f2648a09dd95aef8b9f5968: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1df8519da8b747b48e64cbee84667d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa95d82fe8da48a7a5802d9a197cb479" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37477 } } peers { permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 } } peers { permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 } } }
I20260927 10:53:16.733054 16237 tablet_replica.cc:442] T 8731d83873ae46a9ac29073a3d3e81bb P 21a6a8d06f2648a09dd95aef8b9f5968: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:53:16.739233 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.735123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52250 (local address 127.14.68.1:37477)
0927 10:53:16.735242 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:16.735249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:16.735308 (+ 59us) server_negotiation.cc:407] Connection header received
0927 10:53:16.735461 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:16.735463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:16.735508 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:16.735589 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:16.735881 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.736395 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.737553 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.737887 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.738002 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.738269 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:16.738846 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:16.738962 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:16.739045 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 10:53:16.739102 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:53:16.750061 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.746389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:40513 (local address 127.14.68.3:45303)
0927 10:53:16.746519 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:16.746524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:16.746689 (+ 165us) server_negotiation.cc:407] Connection header received
0927 10:53:16.746774 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:16.746777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:16.746826 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:16.746904 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:16.747126 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.747921 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.748881 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.749209 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.749306 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.749600 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:16.749693 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:16.749765 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:16.749869 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 10:53:16.749926 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:53:16.750052 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.746246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:45303 (local address 127.14.68.1:40513)
0927 10:53:16.746594 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:16.746605 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:16.746700 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:16.746916 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:16.746918 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:16.746923 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:53:16.747022 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.747026 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.748037 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.748040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.748766 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.748776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.749316 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.749319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.749394 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.749437 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:53:16.749776 (+ 339us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:53:16.749781 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 10:53:16.749836 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 10:53:16.749900 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":127,"threads_started":1}
I20260927 10:53:16.756881 16020 raft_consensus.cc:1261] T df37cc5246bc480e9b255a4507706b2e P 21a6a8d06f2648a09dd95aef8b9f5968 [term 1 FOLLOWER]: Refusing update from remote peer fa95d82fe8da48a7a5802d9a197cb479: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:16.757089 16160 raft_consensus.cc:1261] T df37cc5246bc480e9b255a4507706b2e P 1df8519da8b747b48e64cbee84667d83 [term 1 FOLLOWER]: Refusing update from remote peer fa95d82fe8da48a7a5802d9a197cb479: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:16.757355 16260 consensus_queue.cc:1035] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21a6a8d06f2648a09dd95aef8b9f5968" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 32981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:16.757545 16236 consensus_queue.cc:1035] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df8519da8b747b48e64cbee84667d83" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:16.780613 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.777004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:35131 (local address 127.14.68.1:37477)
0927 10:53:16.777175 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:16.777179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:16.777271 (+ 92us) server_negotiation.cc:407] Connection header received
0927 10:53:16.777409 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:16.777411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:16.777450 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:16.777509 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:16.777744 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.778296 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.779363 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.779740 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.779845 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.780216 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:16.780284 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:16.780343 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:16.780470 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 10:53:16.780514 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 10:53:16.780694 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.776896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:37477 (local address 127.14.68.3:35131)
0927 10:53:16.777201 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:16.777217 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:16.777326 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:16.777541 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:16.777544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:16.777552 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:53:16.777653 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.777658 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.778381 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.778385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.779261 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:16.779273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.779928 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.779932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:16.780020 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.780071 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:53:16.780375 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:53:16.780382 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 10:53:16.780455 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 10:53:16.780541 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":83,"threads_started":1}
I20260927 10:53:16.809264 15723 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:16.809445 15723 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:16.815294 15723 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a5e7f5ea3d24567adf8f3b5b272e541: Sending DeleteTablet for 3 replicas of tablet df37cc5246bc480e9b255a4507706b2e
I20260927 10:53:16.816093 15860 tablet_service.cc:1514] Processing DeleteTablet for tablet df37cc5246bc480e9b255a4507706b2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:16 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:52246
I20260927 10:53:16.816371 16252 tablet_replica.cc:330] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479: stopping tablet replica
I20260927 10:53:16.816550 16252 raft_consensus.cc:2229] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:53:16.817009 16252 raft_consensus.cc:2258] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:16.818823 16252 ts_tablet_manager.cc:1905] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:16.819799 16000 tablet_service.cc:1514] Processing DeleteTablet for tablet df37cc5246bc480e9b255a4507706b2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:16 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:35298
I20260927 10:53:16.819973 16253 tablet_replica.cc:330] T df37cc5246bc480e9b255a4507706b2e P 21a6a8d06f2648a09dd95aef8b9f5968: stopping tablet replica
I20260927 10:53:16.820068 16253 raft_consensus.cc:2229] T df37cc5246bc480e9b255a4507706b2e P 21a6a8d06f2648a09dd95aef8b9f5968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:16.820178 16253 raft_consensus.cc:2258] T df37cc5246bc480e9b255a4507706b2e P 21a6a8d06f2648a09dd95aef8b9f5968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:16.821034 16253 ts_tablet_manager.cc:1905] T df37cc5246bc480e9b255a4507706b2e P 21a6a8d06f2648a09dd95aef8b9f5968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:16.822628 16140 tablet_service.cc:1514] Processing DeleteTablet for tablet df37cc5246bc480e9b255a4507706b2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:16 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:53214
I20260927 10:53:16.822825 16254 tablet_replica.cc:330] T df37cc5246bc480e9b255a4507706b2e P 1df8519da8b747b48e64cbee84667d83: stopping tablet replica
I20260927 10:53:16.822959 16254 raft_consensus.cc:2229] T df37cc5246bc480e9b255a4507706b2e P 1df8519da8b747b48e64cbee84667d83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:16.823071 16254 raft_consensus.cc:2258] T df37cc5246bc480e9b255a4507706b2e P 1df8519da8b747b48e64cbee84667d83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:16.823184 16252 ts_tablet_manager.cc:1918] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:53:16.823251 16252 log.cc:1196] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-0/wal/wals/df37cc5246bc480e9b255a4507706b2e
I20260927 10:53:16.823578 16252 ts_tablet_manager.cc:1939] T df37cc5246bc480e9b255a4507706b2e P fa95d82fe8da48a7a5802d9a197cb479: Deleting consensus metadata
I20260927 10:53:16.823851 16254 ts_tablet_manager.cc:1905] T df37cc5246bc480e9b255a4507706b2e P 1df8519da8b747b48e64cbee84667d83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:16.824240 15709 catalog_manager.cc:4841] TS fa95d82fe8da48a7a5802d9a197cb479 (127.14.68.1:37477): tablet df37cc5246bc480e9b255a4507706b2e (table test-table [id=bdce35b41df8445cae1aa138a54abec5]) successfully deleted
I20260927 10:53:16.825417 16253 ts_tablet_manager.cc:1918] T df37cc5246bc480e9b255a4507706b2e P 21a6a8d06f2648a09dd95aef8b9f5968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:53:16.825603 16253 log.cc:1196] T df37cc5246bc480e9b255a4507706b2e P 21a6a8d06f2648a09dd95aef8b9f5968: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/df37cc5246bc480e9b255a4507706b2e
I20260927 10:53:16.825944 16253 ts_tablet_manager.cc:1939] T df37cc5246bc480e9b255a4507706b2e P 21a6a8d06f2648a09dd95aef8b9f5968: Deleting consensus metadata
I20260927 10:53:16.826634 16254 ts_tablet_manager.cc:1918] T df37cc5246bc480e9b255a4507706b2e P 1df8519da8b747b48e64cbee84667d83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:53:16.826713 16254 log.cc:1196] T df37cc5246bc480e9b255a4507706b2e P 1df8519da8b747b48e64cbee84667d83: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/df37cc5246bc480e9b255a4507706b2e
I20260927 10:53:16.826745 15707 catalog_manager.cc:4841] TS 21a6a8d06f2648a09dd95aef8b9f5968 (127.14.68.2:32981): tablet df37cc5246bc480e9b255a4507706b2e (table test-table [id=bdce35b41df8445cae1aa138a54abec5]) successfully deleted
I20260927 10:53:16.826959 16254 ts_tablet_manager.cc:1939] T df37cc5246bc480e9b255a4507706b2e P 1df8519da8b747b48e64cbee84667d83: Deleting consensus metadata
I20260927 10:53:16.827611 15707 catalog_manager.cc:4841] TS 1df8519da8b747b48e64cbee84667d83 (127.14.68.3:45303): tablet df37cc5246bc480e9b255a4507706b2e (table test-table [id=bdce35b41df8445cae1aa138a54abec5]) successfully deleted
Sep 27 10:53:16 dist-test-slave-tz51 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506396, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.68.1@KRBTEST.COM
I20260927 10:53:16.838819 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.827859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52258 (local address 127.14.68.1:37477)
0927 10:53:16.827992 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:16.827997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:16.828445 (+ 448us) server_negotiation.cc:407] Connection header received
0927 10:53:16.828588 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:16.828591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:16.828639 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:16.828732 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:16.829419 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.829925 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.830772 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.831000 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.831109 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.834360 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:16.834372 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:16.834388 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:16.834417 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:16.836606 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:16.837127 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:16.837134 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:16.837136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:16.837259 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:16.837573 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:16.837576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:16.837578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:16.837877 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:16.838106 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:16.838501 (+ 395us) server_negotiation.cc:299] Negotiation successful
0927 10:53:16.838637 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 10:53:16.839803 15920 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52258
I20260927 10:53:16.855391 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:16.846010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41010 (local address 127.14.68.62:43327)
0927 10:53:16.846174 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:16.846185 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:16.846424 (+ 239us) server_negotiation.cc:407] Connection header received
0927 10:53:16.846594 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:16.846598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:16.846647 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:16.846698 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:16.847608 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.848464 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.849624 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:16.849955 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:16.850027 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:16.851912 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:16.851917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:16.851920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:16.851950 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:16.853661 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:16.854136 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:16.854139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:16.854142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:16.854228 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:16.854505 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:16.854508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:16.854510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:16.854726 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:16.854852 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:16.855096 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 10:53:16.855221 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260927 10:53:16.855849 15723 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41010
I20260927 10:53:16.857853 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 15795
I20260927 10:53:16.864917 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 15935
I20260927 10:53:16.871855 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 16075
I20260927 10:53:16.878921 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 15691
2026-09-27T10:53:16Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2974 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:53:16 dist-test-slave-tz51 krb5kdc[16294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:53:16 dist-test-slave-tz51 krb5kdc[16294](info): set up 2 sockets
Sep 27 10:53:16 dist-test-slave-tz51 krb5kdc[16294](info): commencing operation
krb5kdc: starting...
W20260927 10:53:17.903451 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:53:17 dist-test-slave-tz51 krb5kdc[16294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:53:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:53:18.023144 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:39029
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:39029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:18.133985 16307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:18.134266 16307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:18.134334 16307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:18.138317 16307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:18.138398 16307 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 10:53:18.138432 16307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:18.138470 16307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:18.138502 16307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:18.138564 16307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:18.143584 16307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:39029
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:39029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:18.144598 16307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:18.145581 16307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:18.153595 16315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:18.153674 16312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:18.153683 16307 server_base.cc:1034] running on GCE node
W20260927 10:53:18.153617 16313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:18.154417 16307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:18.155419 16307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:18.156598 16307 hybrid_clock.cc:648] HybridClock initialized: now 1790506398156563 us; error 50 us; skew 500 ppm
Sep 27 10:53:18 dist-test-slave-tz51 krb5kdc[16294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506398, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:18.159334 16307 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:53:18.160245 16307 webserver.cc:460] Webserver started at http://127.14.68.62:34719/ using document root <none> and password file <none>
I20260927 10:53:18.160770 16307 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:18.160857 16307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:18.161077 16307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:18.163005 16307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "fc2a9bdceed2423cb802d0803d5964aa"
format_stamp: "Formatted at 2026-09-27 10:53:18 on dist-test-slave-tz51"
server_key: "496971a08b7fb4d3016e03dd1a38a93d"
server_key_iv: "d57498875c9fbe75c5f6d95ca7017194"
server_key_version: "encryptionkey@0"
I20260927 10:53:18.163579 16307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "fc2a9bdceed2423cb802d0803d5964aa"
format_stamp: "Formatted at 2026-09-27 10:53:18 on dist-test-slave-tz51"
server_key: "496971a08b7fb4d3016e03dd1a38a93d"
server_key_iv: "d57498875c9fbe75c5f6d95ca7017194"
server_key_version: "encryptionkey@0"
I20260927 10:53:18.167132 16307 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:53:18.169505 16322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:18.170651 16307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:53:18.170786 16307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "fc2a9bdceed2423cb802d0803d5964aa"
format_stamp: "Formatted at 2026-09-27 10:53:18 on dist-test-slave-tz51"
server_key: "496971a08b7fb4d3016e03dd1a38a93d"
server_key_iv: "d57498875c9fbe75c5f6d95ca7017194"
server_key_version: "encryptionkey@0"
I20260927 10:53:18.170895 16307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:18.187798 16307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:18.190490 16307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:18.190685 16307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:18.198446 16307 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:39029
I20260927 10:53:18.198460 16384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:39029 every 8 connection(s)
I20260927 10:53:18.199311 16307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:18.201905 16385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:18.207041 16385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa: Bootstrap starting.
I20260927 10:53:18.209252 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 16307
I20260927 10:53:18.209267 16385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:18.209378 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:53:18.209646 14608 external_mini_cluster.cc:1428] Setting key 63435b8aa1559ef92b4429f730128317
I20260927 10:53:18.210017 16385 log.cc:824] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:18.211875 16385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa: No bootstrap required, opened a new log
I20260927 10:53:18.215070 16385 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2a9bdceed2423cb802d0803d5964aa" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 39029 } }
I20260927 10:53:18.215349 16385 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:18.215404 16385 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc2a9bdceed2423cb802d0803d5964aa, State: Initialized, Role: FOLLOWER
I20260927 10:53:18.215883 16385 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2a9bdceed2423cb802d0803d5964aa" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 39029 } }
I20260927 10:53:18.216009 16385 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:18.216094 16385 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:18.216215 16385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:18.217236 16385 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2a9bdceed2423cb802d0803d5964aa" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 39029 } }
I20260927 10:53:18.217609 16385 leader_election.cc:302] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc2a9bdceed2423cb802d0803d5964aa; no voters:
I20260927 10:53:18.217971 16385 leader_election.cc:288] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:18.218084 16390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:18.218334 16390 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa [term 1 LEADER]: Becoming Leader. State: Replica: fc2a9bdceed2423cb802d0803d5964aa, State: Running, Role: LEADER
I20260927 10:53:18.218636 16390 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2a9bdceed2423cb802d0803d5964aa" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 39029 } }
Sep 27 10:53:18 dist-test-slave-tz51 krb5kdc[16294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:18.219313 16385 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:18.220386 16391 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc2a9bdceed2423cb802d0803d5964aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2a9bdceed2423cb802d0803d5964aa" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 39029 } } }
I20260927 10:53:18.220518 16391 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:18.220817 16392 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc2a9bdceed2423cb802d0803d5964aa. Latest consensus state: current_term: 1 leader_uuid: "fc2a9bdceed2423cb802d0803d5964aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2a9bdceed2423cb802d0803d5964aa" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 39029 } } }
I20260927 10:53:18.220914 16392 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:18.221622 16399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:18.224829 16399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:18.225350 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:18.210644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33216 (local address 127.14.68.62:39029)
0927 10:53:18.211061 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:18.211080 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:18.211144 (+ 64us) server_negotiation.cc:407] Connection header received
0927 10:53:18.211790 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:18.211806 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:18.212123 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:18.212397 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:18.213266 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.214665 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.215858 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.216504 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.216631 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:18.219578 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:18.219584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:18.219614 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:18.219653 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:18.222155 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:18.222615 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:18.222621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:18.222627 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:18.222770 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:18.223101 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:18.223104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:18.223106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:18.223584 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:18.223817 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:18.224105 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 10:53:18.224393 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":157,"threads_started":1}
I20260927 10:53:18.229991 16399 catalog_manager.cc:1347] Generated new cluster ID: 9132325914964ec0a2ef3e0ab8442433
I20260927 10:53:18.230050 16399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:18.244614 16399 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:18.245462 16399 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:18.249028 16399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa: Generated new TSK 0
I20260927 10:53:18.249650 16399 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
I20260927 10:53:18.282770 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:39029
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:18.394255 16411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:18.394486 16411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:18.394531 16411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:18.398633 16411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:18.398679 16411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:18.398747 16411 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 10:53:18.398780 16411 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:53:18.398860 16411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:53:18.404172 16411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.68.62:39029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:18.405176 16411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:18.406234 16411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:18.415760 16419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:18.415887 16416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:18.415992 16411 server_base.cc:1034] running on GCE node
W20260927 10:53:18.416182 16417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:18.416524 16411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:18.417191 16411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:18.418370 16411 hybrid_clock.cc:648] HybridClock initialized: now 1790506398418358 us; error 24 us; skew 500 ppm
Sep 27 10:53:18 dist-test-slave-tz51 krb5kdc[16294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506398, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:18.421267 16411 init.cc:377] Logged in from keytab as kudu/127.14.68.1@KRBTEST.COM (short username kudu)
I20260927 10:53:18.422112 16411 webserver.cc:460] Webserver started at http://127.14.68.1:45739/ using document root <none> and password file <none>
I20260927 10:53:18.422655 16411 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:18.422741 16411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:18.422963 16411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:18.424952 16411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "5cbbac417b1048d4b0139df810615709"
format_stamp: "Formatted at 2026-09-27 10:53:18 on dist-test-slave-tz51"
server_key: "0c2196343505868d8c2371556a7acb44"
server_key_iv: "19c53785334d790dc0fb9c8ffe05c356"
server_key_version: "encryptionkey@0"
I20260927 10:53:18.425498 16411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "5cbbac417b1048d4b0139df810615709"
format_stamp: "Formatted at 2026-09-27 10:53:18 on dist-test-slave-tz51"
server_key: "0c2196343505868d8c2371556a7acb44"
server_key_iv: "19c53785334d790dc0fb9c8ffe05c356"
server_key_version: "encryptionkey@0"
I20260927 10:53:18.429155 16411 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:53:18.431520 16426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:18.432791 16411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:18.432922 16411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "5cbbac417b1048d4b0139df810615709"
format_stamp: "Formatted at 2026-09-27 10:53:18 on dist-test-slave-tz51"
server_key: "0c2196343505868d8c2371556a7acb44"
server_key_iv: "19c53785334d790dc0fb9c8ffe05c356"
server_key_version: "encryptionkey@0"
I20260927 10:53:18.433032 16411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:18.461126 16411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:18.464514 16411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:18.464720 16411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:18.466137 16411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:18.466209 16411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:18.466274 16411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:18.466324 16411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:53:18 dist-test-slave-tz51 krb5kdc[16294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506398, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:18.477532 16411 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:45141
I20260927 10:53:18.477553 16541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:45141 every 8 connection(s)
I20260927 10:53:18.478511 16411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:53:18.478875 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 16411
I20260927 10:53:18.479000 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:53:18.479317 14608 external_mini_cluster.cc:1428] Setting key 260bbc1e1f2faca7a6095b7f4050e16e
I20260927 10:53:18.481149 16435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:18.467531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:39029 (local address 127.14.68.1:58319)
0927 10:53:18.468031 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:18.468072 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:18.469019 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:18.469769 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:18.469775 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:18.470113 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:18.470555 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:18.470568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.471528 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.471533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:18.472480 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:18.472489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.472988 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.472992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:18.473219 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:18.473824 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:18.473865 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:18.476252 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:18.478137 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:18.478145 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:18.478161 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:18.478480 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:18.478813 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:18.478818 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:18.478820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:18.478995 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:18.479530 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:18.479534 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:18.479812 (+ 278us) client_negotiation.cc:769] Sending connection context
0927 10:53:18.479995 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 10:53:18.480315 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":155,"threads_started":1}
I20260927 10:53:18.481959 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:18.466992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:58319 (local address 127.14.68.62:39029)
0927 10:53:18.467120 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:18.467124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:18.468189 (+ 1065us) server_negotiation.cc:407] Connection header received
0927 10:53:18.469271 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:18.469275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:18.469357 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:18.469420 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:18.470713 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.471399 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.472609 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.472839 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.472951 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:18.476433 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:18.476438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:18.476441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:18.476469 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:18.478025 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:18.478618 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:18.478622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:18.478624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:18.478721 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:18.479092 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:18.479098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:18.479100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:18.479354 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:18.479476 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:18.481747 (+ 2271us) server_negotiation.cc:299] Negotiation successful
0927 10:53:18.481846 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:53:18.483009 16542 heartbeater.cc:347] Connected to a master server at 127.14.68.62:39029
I20260927 10:53:18.483174 16542 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:18.483649 16542 heartbeater.cc:511] Master 127.14.68.62:39029 requested a full tablet report, sending...
I20260927 10:53:18.485235 16339 ts_manager.cc:194] Registered new tserver with Master: 5cbbac417b1048d4b0139df810615709 (127.14.68.1:45141)
I20260927 10:53:18.486883 16339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:58319
WARNING: no policy specified for kudu/127.14.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.2@KRBTEST.COM" created.
I20260927 10:53:18.494828 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:18.487889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:43031 (local address 127.14.68.62:39029)
0927 10:53:18.487988 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:18.487992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:18.488285 (+ 293us) server_negotiation.cc:407] Connection header received
0927 10:53:18.488350 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:18.488353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:18.488397 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:18.488477 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:18.488865 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.489599 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.490385 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.490760 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.490866 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:18.491966 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:18.491972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:18.491978 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:18.492005 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:18.493304 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:18.493705 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:18.493708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:18.493709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:18.493777 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:18.494022 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:18.494026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:18.494028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:18.494186 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:18.494318 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:18.494584 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 10:53:18.494683 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 10:53:18.494839 16549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:18.487849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:39029 (local address 127.14.68.1:43031)
0927 10:53:18.488196 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:18.488209 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:18.488287 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:18.488496 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:18.488498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:18.488669 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:18.488756 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:18.488762 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.489731 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.489733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:18.490269 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:18.490276 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.490879 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.490881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:18.490945 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:18.491456 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:18.491468 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:18.491837 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:18.493417 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:18.493420 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:18.493421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:18.493618 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:18.493857 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:18.493859 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:18.493860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:18.493931 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:18.494320 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:18.494323 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:18.494425 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 10:53:18.494524 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 10:53:18.494663 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":198,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
I20260927 10:53:18.508678 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:39029
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:18.613489 16551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:18.613719 16551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:18.613765 16551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:18.617735 16551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:18.617784 16551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:18.617823 16551 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 10:53:18.617875 16551 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:53:18.617923 16551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:53:18.623354 16551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.68.62:39029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:18.624310 16551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:18.625253 16551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:18.634351 16559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:18.634378 16556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:18.634686 16557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:18.635130 16551 server_base.cc:1034] running on GCE node
I20260927 10:53:18.635515 16551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:18.636170 16551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:18.637379 16551 hybrid_clock.cc:648] HybridClock initialized: now 1790506398637360 us; error 28 us; skew 500 ppm
Sep 27 10:53:18 dist-test-slave-tz51 krb5kdc[16294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506398, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:18.640094 16551 init.cc:377] Logged in from keytab as kudu/127.14.68.2@KRBTEST.COM (short username kudu)
I20260927 10:53:18.640847 16551 webserver.cc:460] Webserver started at http://127.14.68.2:35203/ using document root <none> and password file <none>
I20260927 10:53:18.641328 16551 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:18.641383 16551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:18.641534 16551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:18.643426 16551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "e6f4bc6c7c4d4b9583c043f619299d9c"
format_stamp: "Formatted at 2026-09-27 10:53:18 on dist-test-slave-tz51"
server_key: "7320ec89b490c58fe7d77eb60244cc4e"
server_key_iv: "20c0abf73fa30c84025d9007e8e0aa3d"
server_key_version: "encryptionkey@0"
I20260927 10:53:18.643982 16551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "e6f4bc6c7c4d4b9583c043f619299d9c"
format_stamp: "Formatted at 2026-09-27 10:53:18 on dist-test-slave-tz51"
server_key: "7320ec89b490c58fe7d77eb60244cc4e"
server_key_iv: "20c0abf73fa30c84025d9007e8e0aa3d"
server_key_version: "encryptionkey@0"
I20260927 10:53:18.647420 16551 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:53:18.649698 16566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:18.650888 16551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:18.650979 16551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "e6f4bc6c7c4d4b9583c043f619299d9c"
format_stamp: "Formatted at 2026-09-27 10:53:18 on dist-test-slave-tz51"
server_key: "7320ec89b490c58fe7d77eb60244cc4e"
server_key_iv: "20c0abf73fa30c84025d9007e8e0aa3d"
server_key_version: "encryptionkey@0"
I20260927 10:53:18.651110 16551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:18.674242 16551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:18.677158 16551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:18.677368 16551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:18.678753 16551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:18.678824 16551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:18.678893 16551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:18.678941 16551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:53:18 dist-test-slave-tz51 krb5kdc[16294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506398, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:18.689754 16551 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:37707
I20260927 10:53:18.689817 16681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:37707 every 8 connection(s)
I20260927 10:53:18.690704 16551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:53:18.692431 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:18.679670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:51657 (local address 127.14.68.62:39029)
0927 10:53:18.679811 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:18.679815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:18.680862 (+ 1047us) server_negotiation.cc:407] Connection header received
0927 10:53:18.681987 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:18.681990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:18.682050 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:18.682159 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:18.683779 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.684417 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.685508 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.685747 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.685859 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:18.688782 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:18.688786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:18.688789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:18.688815 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:18.690377 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:18.690923 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:18.690925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:18.690926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:18.691000 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:18.691461 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:18.691467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:18.691469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:18.691727 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:18.691826 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:18.692223 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 10:53:18.692317 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:53:18.693166 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:18.680256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:39029 (local address 127.14.68.2:51657)
0927 10:53:18.680721 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:18.680761 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:18.681725 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:18.682368 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:18.682379 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:18.682846 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:18.683548 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:18.683571 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.684534 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.684537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:18.685384 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:18.685393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.685905 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.685909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:18.686090 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:18.686668 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:18.686715 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:18.688602 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:18.690486 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:18.690491 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:18.690503 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:18.690831 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:18.691096 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:18.691102 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:18.691105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:18.691314 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:18.691818 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:18.691822 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:18.692022 (+ 200us) client_negotiation.cc:769] Sending connection context
0927 10:53:18.692170 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 10:53:18.692390 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":133,"threads_started":1}
I20260927 10:53:18.694151 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 16551
I20260927 10:53:18.694288 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:53:18.694407 16682 heartbeater.cc:347] Connected to a master server at 127.14.68.62:39029
I20260927 10:53:18.694561 16682 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:18.694587 14608 external_mini_cluster.cc:1428] Setting key 590ac6a39ebaefa5cdfd549c286ee664
I20260927 10:53:18.695147 16682 heartbeater.cc:511] Master 127.14.68.62:39029 requested a full tablet report, sending...
I20260927 10:53:18.696167 16338 ts_manager.cc:194] Registered new tserver with Master: e6f4bc6c7c4d4b9583c043f619299d9c (127.14.68.2:37707)
I20260927 10:53:18.696849 16338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:51657
I20260927 10:53:18.703882 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:18.696683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:48913 (local address 127.14.68.62:39029)
0927 10:53:18.696804 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:18.696810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:18.697251 (+ 441us) server_negotiation.cc:407] Connection header received
0927 10:53:18.697354 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:18.697360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:18.697413 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:18.697495 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:18.697962 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.698431 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.699338 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.699547 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.699651 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:18.700779 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:18.700782 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:18.700785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:18.700807 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:18.702222 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:18.702702 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:18.702706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:18.702708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:18.702782 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:18.703049 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:18.703054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:18.703056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:18.703253 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:18.703363 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:18.703616 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 10:53:18.703742 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:53:18.703891 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:18.696596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:39029 (local address 127.14.68.2:48913)
0927 10:53:18.697157 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:18.697171 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:18.697268 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:18.697523 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:18.697526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:18.697729 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:18.697838 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:18.697845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.698554 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.698557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:18.699230 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:18.699247 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.699654 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.699658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:18.699776 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:18.700251 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:18.700270 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:18.700633 (+ 363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:18.702325 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:18.702330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:18.702332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:18.702597 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:18.702875 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:18.702878 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:18.702880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:18.702961 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:18.703363 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:18.703367 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:18.703454 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 10:53:18.703555 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 10:53:18.703689 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.14.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
I20260927 10:53:18.722033 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:39029
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:18.835578 16691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:18.835875 16691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:18.835975 16691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:18.840049 16691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:18.840095 16691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:18.840154 16691 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 10:53:18.840186 16691 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:53:18.840272 16691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:53:18.845423 16691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.68.62:39029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:18.846397 16691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:18.847342 16691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:18.856633 16699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:18.856650 16696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:18.856711 16691 server_base.cc:1034] running on GCE node
W20260927 10:53:18.856657 16697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:18.857266 16691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:18.857916 16691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:18.859102 16691 hybrid_clock.cc:648] HybridClock initialized: now 1790506398859076 us; error 42 us; skew 500 ppm
Sep 27 10:53:18 dist-test-slave-tz51 krb5kdc[16294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506398, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:18.861863 16691 init.cc:377] Logged in from keytab as kudu/127.14.68.3@KRBTEST.COM (short username kudu)
I20260927 10:53:18.862705 16691 webserver.cc:460] Webserver started at http://127.14.68.3:33201/ using document root <none> and password file <none>
I20260927 10:53:18.863305 16691 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:18.863394 16691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:18.863623 16691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:18.865543 16691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "de1e8de15ff04a93a0cfdc98e9312d70"
format_stamp: "Formatted at 2026-09-27 10:53:18 on dist-test-slave-tz51"
server_key: "162a1dc3e16c09d37d3836f45276292c"
server_key_iv: "b900aaaf80f527b3e595fcc124df7c4b"
server_key_version: "encryptionkey@0"
I20260927 10:53:18.866137 16691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "de1e8de15ff04a93a0cfdc98e9312d70"
format_stamp: "Formatted at 2026-09-27 10:53:18 on dist-test-slave-tz51"
server_key: "162a1dc3e16c09d37d3836f45276292c"
server_key_iv: "b900aaaf80f527b3e595fcc124df7c4b"
server_key_version: "encryptionkey@0"
I20260927 10:53:18.870151 16691 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 10:53:18.872530 16706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:18.873806 16691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:18.873944 16691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "de1e8de15ff04a93a0cfdc98e9312d70"
format_stamp: "Formatted at 2026-09-27 10:53:18 on dist-test-slave-tz51"
server_key: "162a1dc3e16c09d37d3836f45276292c"
server_key_iv: "b900aaaf80f527b3e595fcc124df7c4b"
server_key_version: "encryptionkey@0"
I20260927 10:53:18.874063 16691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:18.895618 16691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:18.898298 16691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:18.898486 16691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:18.900018 16691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:18.900092 16691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:18.900161 16691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:18.900195 16691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:53:18 dist-test-slave-tz51 krb5kdc[16294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506398, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:18.910873 16691 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:46341
I20260927 10:53:18.910950 16821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:46341 every 8 connection(s)
I20260927 10:53:18.911746 16691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:53:18.914490 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:18.900800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:51263 (local address 127.14.68.62:39029)
0927 10:53:18.900951 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:18.900958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:18.901782 (+ 824us) server_negotiation.cc:407] Connection header received
0927 10:53:18.903017 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:18.903023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:18.903156 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:18.903226 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:18.905051 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.905963 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.907245 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.907482 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.907596 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:18.910988 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:18.910992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:18.910995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:18.911023 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:18.912684 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:18.913142 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:18.913144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:18.913145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:18.913214 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:18.913539 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:18.913542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:18.913543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:18.913701 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:18.913774 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:18.914213 (+ 439us) server_negotiation.cc:299] Negotiation successful
0927 10:53:18.914328 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:53:18.915176 16715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:18.901146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:39029 (local address 127.14.68.3:51263)
0927 10:53:18.901612 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:18.901668 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:18.902759 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:18.903664 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:18.903674 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:18.904190 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:18.904862 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:18.904887 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.906095 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.906100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:18.907127 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:18.907136 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.907665 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.907670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:18.907864 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:18.909045 (+ 1181us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:18.909084 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:18.910782 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:18.912791 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:18.912795 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:18.912804 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:18.913034 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:18.913322 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:18.913324 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:18.913326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:18.913455 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:18.913788 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:18.913792 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:18.913991 (+ 199us) client_negotiation.cc:769] Sending connection context
0927 10:53:18.914145 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 10:53:18.914365 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":104,"threads_started":1}
I20260927 10:53:18.916291 16822 heartbeater.cc:347] Connected to a master server at 127.14.68.62:39029
I20260927 10:53:18.916445 16822 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:18.916903 16822 heartbeater.cc:511] Master 127.14.68.62:39029 requested a full tablet report, sending...
I20260927 10:53:18.917886 16339 ts_manager.cc:194] Registered new tserver with Master: de1e8de15ff04a93a0cfdc98e9312d70 (127.14.68.3:46341)
I20260927 10:53:18.918541 16339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:51263
I20260927 10:53:18.921165 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 16691
I20260927 10:53:18.921277 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:53:18.921610 14608 external_mini_cluster.cc:1428] Setting key 3c0037e9cb4623f957121cde785c0306
I20260927 10:53:18.923789 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:53:18.925482 16828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:18.918316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:39029 (local address 127.14.68.3:59885)
0927 10:53:18.918629 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:18.918662 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:18.918756 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:18.919012 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:18.919015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:18.919231 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:18.919331 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:18.919337 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.920251 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.920255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:18.920892 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:18.920900 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.921377 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.921381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:18.921487 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:18.921945 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:18.921962 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:18.922290 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:18.924151 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:18.924154 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:18.924155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:18.924336 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:18.924623 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:18.924625 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:18.924626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:18.924695 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:18.925116 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:18.925118 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:18.925184 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 10:53:18.925263 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 10:53:18.925358 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":1,"thread_start_us":96,"threads_started":1}
I20260927 10:53:18.925698 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:18.918382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:59885 (local address 127.14.68.62:39029)
0927 10:53:18.918498 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:18.918504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:18.918735 (+ 231us) server_negotiation.cc:407] Connection header received
0927 10:53:18.918846 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:18.918849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:18.918890 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:18.918981 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:18.919448 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.920126 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.921004 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.921231 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.921342 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:18.922439 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:18.922442 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:18.922445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:18.922469 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:18.924034 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:18.924431 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:18.924435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:18.924436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:18.924530 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:18.924781 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:18.924787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:18.924792 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:18.925024 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:18.925133 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:18.925344 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 10:53:18.925468 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 10:53:18 dist-test-slave-tz51 krb5kdc[16294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:53:18 dist-test-slave-tz51 krb5kdc[16294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:18.948797 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:18.941086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33232 (local address 127.14.68.62:39029)
0927 10:53:18.941216 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:18.941220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:18.941230 (+ 10us) server_negotiation.cc:407] Connection header received
0927 10:53:18.941258 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:18.941260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:18.941292 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:18.941378 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:18.942126 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.942610 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.943422 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.943633 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.943756 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:18.945953 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:18.945959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:18.945966 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:18.945994 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:18.947176 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:18.947564 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:18.947570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:18.947572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:18.947648 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:18.948005 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:18.948008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:18.948010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:18.948177 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:18.948307 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:18.948536 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 10:53:18.948641 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:53:18.951493 16339 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:53:18.954530 16339 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:18.968120 16841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:18.962879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56228 (local address 127.14.68.1:45141)
0927 10:53:18.963149 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:18.963163 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:18.963659 (+ 496us) server_negotiation.cc:407] Connection header received
0927 10:53:18.963778 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:18.963783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:18.963991 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:18.964214 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:18.964464 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.965542 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.966908 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.967441 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.967564 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:18.967765 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:18.967881 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 10:53:18.967970 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":83,"threads_started":1}
I20260927 10:53:18.968106 16839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:18.962739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:45141 (local address 127.0.0.1:56228)
0927 10:53:18.963323 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:18.963373 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:18.963524 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:18.964208 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:18.964212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:18.964233 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:18.964349 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:18.964353 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.965648 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.965654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:18.966794 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:18.966803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.967563 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.967567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:18.967668 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:18.967729 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 10:53:18.967847 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 10:53:18.967966 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":95,"threads_started":1}
I20260927 10:53:18.969596 16840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:18.962798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:37707 (local address 127.0.0.1:41102)
0927 10:53:18.964013 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:18.964027 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:18.964114 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:18.964972 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:18.964975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:18.964992 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:18.965104 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:18.965114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.966597 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.966601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:18.967808 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:18.967818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.969190 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.969194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:18.969309 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:18.969332 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:53:18.969407 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 10:53:18.969473 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":17,"thread_start_us":92,"threads_started":1}
I20260927 10:53:18.969868 16844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:18.963871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41102 (local address 127.14.68.2:37707)
0927 10:53:18.964584 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:18.964596 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:18.964618 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:53:18.964700 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:18.964706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:18.964842 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:18.964994 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:18.965235 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.966444 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.967928 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.969062 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.969178 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:18.969340 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:18.969435 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 10:53:18.969533 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":110,"threads_started":1}
I20260927 10:53:18.969940 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:18.963479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:46341 (local address 127.0.0.1:57128)
0927 10:53:18.964474 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:18.964487 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:18.964584 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:18.965152 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:18.965155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:18.965172 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:18.965247 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:18.965253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.966614 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.966617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:18.968337 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:18.968347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.969543 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.969547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:18.969633 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:18.969655 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:53:18.969735 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 10:53:18.969798 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":62,"thread_start_us":79,"threads_started":1}
I20260927 10:53:18.970381 16843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:18.963664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.14.68.3:46341)
0927 10:53:18.964366 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:18.964378 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:18.964586 (+ 208us) server_negotiation.cc:407] Connection header received
0927 10:53:18.964690 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:18.964695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:18.964876 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:18.965015 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:18.965366 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.966498 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.968466 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:18.969227 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:18.969330 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:18.969483 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:18.970162 (+ 679us) server_negotiation.cc:299] Negotiation successful
0927 10:53:18.970262 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":117,"threads_started":1}
I20260927 10:53:18.971165 16476 tablet_service.cc:1467] Processing CreateTablet for tablet ddc7d13e0a2f4516880e5eb797ac79fc (DEFAULT_TABLE table=test-table [id=7aea5ab12c564219b0eeb96745809305]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:18.972328 16476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc7d13e0a2f4516880e5eb797ac79fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:18.972949 16756 tablet_service.cc:1467] Processing CreateTablet for tablet ddc7d13e0a2f4516880e5eb797ac79fc (DEFAULT_TABLE table=test-table [id=7aea5ab12c564219b0eeb96745809305]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:18.973274 16616 tablet_service.cc:1467] Processing CreateTablet for tablet ddc7d13e0a2f4516880e5eb797ac79fc (DEFAULT_TABLE table=test-table [id=7aea5ab12c564219b0eeb96745809305]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:18.973951 16756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc7d13e0a2f4516880e5eb797ac79fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:18.974288 16616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc7d13e0a2f4516880e5eb797ac79fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:18.980825 16845 tablet_bootstrap.cc:492] T ddc7d13e0a2f4516880e5eb797ac79fc P 5cbbac417b1048d4b0139df810615709: Bootstrap starting.
I20260927 10:53:18.981232 16846 tablet_bootstrap.cc:492] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c: Bootstrap starting.
I20260927 10:53:18.981529 16847 tablet_bootstrap.cc:492] T ddc7d13e0a2f4516880e5eb797ac79fc P de1e8de15ff04a93a0cfdc98e9312d70: Bootstrap starting.
I20260927 10:53:18.983937 16845 tablet_bootstrap.cc:654] T ddc7d13e0a2f4516880e5eb797ac79fc P 5cbbac417b1048d4b0139df810615709: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:18.983935 16847 tablet_bootstrap.cc:654] T ddc7d13e0a2f4516880e5eb797ac79fc P de1e8de15ff04a93a0cfdc98e9312d70: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:18.984813 16847 log.cc:824] T ddc7d13e0a2f4516880e5eb797ac79fc P de1e8de15ff04a93a0cfdc98e9312d70: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:18.984812 16845 log.cc:824] T ddc7d13e0a2f4516880e5eb797ac79fc P 5cbbac417b1048d4b0139df810615709: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:18.985621 16846 tablet_bootstrap.cc:654] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:18.986284 16846 log.cc:824] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:18.986825 16847 tablet_bootstrap.cc:492] T ddc7d13e0a2f4516880e5eb797ac79fc P de1e8de15ff04a93a0cfdc98e9312d70: No bootstrap required, opened a new log
I20260927 10:53:18.986847 16845 tablet_bootstrap.cc:492] T ddc7d13e0a2f4516880e5eb797ac79fc P 5cbbac417b1048d4b0139df810615709: No bootstrap required, opened a new log
I20260927 10:53:18.987017 16847 ts_tablet_manager.cc:1397] T ddc7d13e0a2f4516880e5eb797ac79fc P de1e8de15ff04a93a0cfdc98e9312d70: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:53:18.987017 16845 ts_tablet_manager.cc:1397] T ddc7d13e0a2f4516880e5eb797ac79fc P 5cbbac417b1048d4b0139df810615709: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:53:18.987994 16846 tablet_bootstrap.cc:492] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c: No bootstrap required, opened a new log
I20260927 10:53:18.988189 16846 ts_tablet_manager.cc:1397] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20260927 10:53:18.990924 16845 raft_consensus.cc:348] T ddc7d13e0a2f4516880e5eb797ac79fc P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:18.990924 16847 raft_consensus.cc:348] T ddc7d13e0a2f4516880e5eb797ac79fc P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:18.991189 16845 raft_consensus.cc:374] T ddc7d13e0a2f4516880e5eb797ac79fc P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:18.991189 16847 raft_consensus.cc:374] T ddc7d13e0a2f4516880e5eb797ac79fc P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:18.991242 16845 raft_consensus.cc:729] T ddc7d13e0a2f4516880e5eb797ac79fc P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cbbac417b1048d4b0139df810615709, State: Initialized, Role: FOLLOWER
I20260927 10:53:18.991250 16847 raft_consensus.cc:729] T ddc7d13e0a2f4516880e5eb797ac79fc P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de1e8de15ff04a93a0cfdc98e9312d70, State: Initialized, Role: FOLLOWER
I20260927 10:53:18.991083 16846 raft_consensus.cc:348] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:18.991457 16846 raft_consensus.cc:374] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:18.991524 16846 raft_consensus.cc:729] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f4bc6c7c4d4b9583c043f619299d9c, State: Initialized, Role: FOLLOWER
I20260927 10:53:18.991737 16847 consensus_queue.cc:260] T ddc7d13e0a2f4516880e5eb797ac79fc P de1e8de15ff04a93a0cfdc98e9312d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:18.991737 16845 consensus_queue.cc:260] T ddc7d13e0a2f4516880e5eb797ac79fc P 5cbbac417b1048d4b0139df810615709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:18.991981 16846 consensus_queue.cc:260] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:18.992537 16822 heartbeater.cc:503] Master 127.14.68.62:39029 was elected leader, sending a full tablet report...
I20260927 10:53:18.992765 16542 heartbeater.cc:503] Master 127.14.68.62:39029 was elected leader, sending a full tablet report...
I20260927 10:53:18.992856 16845 ts_tablet_manager.cc:1428] T ddc7d13e0a2f4516880e5eb797ac79fc P 5cbbac417b1048d4b0139df810615709: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260927 10:53:18.992856 16847 ts_tablet_manager.cc:1428] T ddc7d13e0a2f4516880e5eb797ac79fc P de1e8de15ff04a93a0cfdc98e9312d70: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:53:18.994019 16682 heartbeater.cc:503] Master 127.14.68.62:39029 was elected leader, sending a full tablet report...
I20260927 10:53:18.995210 16846 ts_tablet_manager.cc:1428] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
I20260927 10:53:19.128764 16853 raft_consensus.cc:482] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:19.128991 16853 raft_consensus.cc:504] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:19.130177 16853 leader_election.cc:288] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cbbac417b1048d4b0139df810615709 (127.14.68.1:45141), de1e8de15ff04a93a0cfdc98e9312d70 (127.14.68.3:46341)
I20260927 10:53:19.133764 16841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:19.130541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:52371 (local address 127.14.68.1:45141)
0927 10:53:19.130676 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:19.130682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:19.130702 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:53:19.130832 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:19.130835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:19.130881 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:19.130964 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:19.131251 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.131823 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.132802 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.133224 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.133344 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:19.133414 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:19.133574 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 10:53:19.133645 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:53:19.133764 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:19.130412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:45141 (local address 127.14.68.2:52371)
0927 10:53:19.130581 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:19.130597 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:19.130691 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:19.130987 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:19.130990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:19.131035 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:19.131143 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:19.131148 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.131928 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.131934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:19.132682 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:19.132690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.133343 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.133347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:19.133448 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:19.133473 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:53:19.133566 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 10:53:19.133635 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"spinlock_wait_cycles":12032}
I20260927 10:53:19.134421 16843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:19.130589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:38207 (local address 127.14.68.3:46341)
0927 10:53:19.130726 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:19.130732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:19.130975 (+ 243us) server_negotiation.cc:407] Connection header received
0927 10:53:19.131141 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:19.131145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:19.131196 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:19.131283 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:19.131926 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.132400 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.133376 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.133774 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.133901 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:19.133974 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:19.134266 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 10:53:19.134326 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:53:19.134423 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:19.130428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:46341 (local address 127.14.68.2:38207)
0927 10:53:19.130878 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:19.130889 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:19.131036 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:19.131292 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:19.131295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:19.131309 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:19.131440 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:19.131445 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.132508 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.132514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:19.133254 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:19.133261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.134002 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.134009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:19.134108 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:19.134124 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 10:53:19.134220 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 10:53:19.134280 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":110,"spinlock_wait_cycles":18176,"thread_start_us":125,"threads_started":1}
I20260927 10:53:19.134447 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc7d13e0a2f4516880e5eb797ac79fc" candidate_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cbbac417b1048d4b0139df810615709" is_pre_election: true
I20260927 10:53:19.134764 16496 raft_consensus.cc:2454] T ddc7d13e0a2f4516880e5eb797ac79fc P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6f4bc6c7c4d4b9583c043f619299d9c in term 0.
I20260927 10:53:19.135079 16776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc7d13e0a2f4516880e5eb797ac79fc" candidate_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" is_pre_election: true
I20260927 10:53:19.135294 16569 leader_election.cc:302] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cbbac417b1048d4b0139df810615709, e6f4bc6c7c4d4b9583c043f619299d9c; no voters:
I20260927 10:53:19.135340 16776 raft_consensus.cc:2454] T ddc7d13e0a2f4516880e5eb797ac79fc P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6f4bc6c7c4d4b9583c043f619299d9c in term 0.
I20260927 10:53:19.135550 16853 raft_consensus.cc:2781] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:19.135635 16853 raft_consensus.cc:482] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:19.135685 16853 raft_consensus.cc:3037] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:19.136821 16853 raft_consensus.cc:504] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:19.137182 16853 leader_election.cc:288] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c [CANDIDATE]: Term 1 election: Requested vote from peers 5cbbac417b1048d4b0139df810615709 (127.14.68.1:45141), de1e8de15ff04a93a0cfdc98e9312d70 (127.14.68.3:46341)
I20260927 10:53:19.137483 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc7d13e0a2f4516880e5eb797ac79fc" candidate_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cbbac417b1048d4b0139df810615709"
I20260927 10:53:19.137532 16776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc7d13e0a2f4516880e5eb797ac79fc" candidate_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de1e8de15ff04a93a0cfdc98e9312d70"
I20260927 10:53:19.137645 16496 raft_consensus.cc:3037] T ddc7d13e0a2f4516880e5eb797ac79fc P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:19.137657 16776 raft_consensus.cc:3037] T ddc7d13e0a2f4516880e5eb797ac79fc P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:19.138625 16776 raft_consensus.cc:2454] T ddc7d13e0a2f4516880e5eb797ac79fc P de1e8de15ff04a93a0cfdc98e9312d70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6f4bc6c7c4d4b9583c043f619299d9c in term 1.
I20260927 10:53:19.138633 16496 raft_consensus.cc:2454] T ddc7d13e0a2f4516880e5eb797ac79fc P 5cbbac417b1048d4b0139df810615709 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6f4bc6c7c4d4b9583c043f619299d9c in term 1.
I20260927 10:53:19.138921 16568 leader_election.cc:302] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de1e8de15ff04a93a0cfdc98e9312d70, e6f4bc6c7c4d4b9583c043f619299d9c; no voters:
I20260927 10:53:19.139120 16853 raft_consensus.cc:2781] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:19.139349 16853 raft_consensus.cc:686] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c [term 1 LEADER]: Becoming Leader. State: Replica: e6f4bc6c7c4d4b9583c043f619299d9c, State: Running, Role: LEADER
I20260927 10:53:19.139667 16853 consensus_queue.cc:237] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:19.142927 16338 catalog_manager.cc:5495] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c reported cstate change: term changed from 0 to 1, leader changed from <none> to e6f4bc6c7c4d4b9583c043f619299d9c (127.14.68.2). New cstate: current_term: 1 leader_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } health_report { overall_health: UNKNOWN } } }
W20260927 10:53:19.162964 16823 tablet.cc:2369] T ddc7d13e0a2f4516880e5eb797ac79fc P de1e8de15ff04a93a0cfdc98e9312d70: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:53:19.192049 16683 tablet.cc:2369] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 10:53:19 dist-test-slave-tz51 krb5kdc[16294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506398, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:19.202418 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:19.193790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33236 (local address 127.14.68.62:39029)
0927 10:53:19.193985 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:19.193992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:19.194237 (+ 245us) server_negotiation.cc:407] Connection header received
0927 10:53:19.194422 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:19.194426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:19.194473 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:19.194641 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:19.195328 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.196036 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.196825 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.197024 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.197133 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:19.199265 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:19.199269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:19.199270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:19.199289 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:19.200748 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:19.201181 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:19.201187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:19.201189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:19.201273 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:19.201592 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:19.201595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:19.201596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:19.201752 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:19.201825 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:19.202192 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 10:53:19.202291 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":80}
I20260927 10:53:19.202420 16862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:19.193590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:39029 (local address 127.0.0.1:33236)
0927 10:53:19.194146 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:19.194158 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:19.194318 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:19.194690 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:19.194692 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:19.195082 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:19.195191 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:19.195197 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.196175 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.196177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:19.196727 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:19.196735 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.197140 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.197143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:19.197242 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:19.197746 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:19.197757 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:19.199105 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:19.200854 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:19.200865 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:19.200878 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:19.201080 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:19.201393 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:19.201396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:19.201399 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:19.201502 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:19.201852 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:19.201858 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:19.201972 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 10:53:19.202084 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 10:53:19.202220 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":29,"thread_start_us":138,"threads_started":1}
I20260927 10:53:19.206542 16338 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:33236:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 10:53:19.207260 16338 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:19.211973 16476 tablet_service.cc:1467] Processing CreateTablet for tablet 6e821a7d79e44edebb2b2be52fb03a22 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=45811bbbe35a48b6858a25bca547bb72]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:53:19.212214 16616 tablet_service.cc:1467] Processing CreateTablet for tablet 6e821a7d79e44edebb2b2be52fb03a22 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=45811bbbe35a48b6858a25bca547bb72]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:53:19.212286 16756 tablet_service.cc:1467] Processing CreateTablet for tablet 6e821a7d79e44edebb2b2be52fb03a22 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=45811bbbe35a48b6858a25bca547bb72]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:53:19.212729 16476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e821a7d79e44edebb2b2be52fb03a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:19.212764 16616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e821a7d79e44edebb2b2be52fb03a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:19.212935 16756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e821a7d79e44edebb2b2be52fb03a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:19.215234 16847 tablet_bootstrap.cc:492] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70: Bootstrap starting.
I20260927 10:53:19.215516 16845 tablet_bootstrap.cc:492] T 6e821a7d79e44edebb2b2be52fb03a22 P 5cbbac417b1048d4b0139df810615709: Bootstrap starting.
I20260927 10:53:19.216282 16847 tablet_bootstrap.cc:654] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:19.216504 16845 tablet_bootstrap.cc:654] T 6e821a7d79e44edebb2b2be52fb03a22 P 5cbbac417b1048d4b0139df810615709: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:19.217170 16846 tablet_bootstrap.cc:492] T 6e821a7d79e44edebb2b2be52fb03a22 P e6f4bc6c7c4d4b9583c043f619299d9c: Bootstrap starting.
I20260927 10:53:19.217473 16847 tablet_bootstrap.cc:492] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70: No bootstrap required, opened a new log
I20260927 10:53:19.217545 16847 ts_tablet_manager.cc:1397] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 10:53:19.217686 16845 tablet_bootstrap.cc:492] T 6e821a7d79e44edebb2b2be52fb03a22 P 5cbbac417b1048d4b0139df810615709: No bootstrap required, opened a new log
I20260927 10:53:19.217744 16845 ts_tablet_manager.cc:1397] T 6e821a7d79e44edebb2b2be52fb03a22 P 5cbbac417b1048d4b0139df810615709: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:19.217971 16847 raft_consensus.cc:348] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:19.218110 16846 tablet_bootstrap.cc:654] T 6e821a7d79e44edebb2b2be52fb03a22 P e6f4bc6c7c4d4b9583c043f619299d9c: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:19.218096 16847 raft_consensus.cc:374] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:19.218144 16847 raft_consensus.cc:729] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de1e8de15ff04a93a0cfdc98e9312d70, State: Initialized, Role: FOLLOWER
I20260927 10:53:19.218133 16845 raft_consensus.cc:348] T 6e821a7d79e44edebb2b2be52fb03a22 P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:19.218247 16845 raft_consensus.cc:374] T 6e821a7d79e44edebb2b2be52fb03a22 P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:19.218283 16845 raft_consensus.cc:729] T 6e821a7d79e44edebb2b2be52fb03a22 P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cbbac417b1048d4b0139df810615709, State: Initialized, Role: FOLLOWER
I20260927 10:53:19.218273 16847 consensus_queue.cc:260] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:19.218403 16845 consensus_queue.cc:260] T 6e821a7d79e44edebb2b2be52fb03a22 P 5cbbac417b1048d4b0139df810615709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:19.218678 16845 ts_tablet_manager.cc:1428] T 6e821a7d79e44edebb2b2be52fb03a22 P 5cbbac417b1048d4b0139df810615709: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:53:19.218679 16851 tablet_replica.cc:439] T 6e821a7d79e44edebb2b2be52fb03a22 P 5cbbac417b1048d4b0139df810615709: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } } }
I20260927 10:53:19.218886 16851 tablet_replica.cc:442] T 6e821a7d79e44edebb2b2be52fb03a22 P 5cbbac417b1048d4b0139df810615709: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:53:19.218873 16852 tablet_replica.cc:439] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } } }
I20260927 10:53:19.218998 16852 tablet_replica.cc:442] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:53:19.219221 16847 ts_tablet_manager.cc:1428] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 10:53:19.220139 16846 tablet_bootstrap.cc:492] T 6e821a7d79e44edebb2b2be52fb03a22 P e6f4bc6c7c4d4b9583c043f619299d9c: No bootstrap required, opened a new log
I20260927 10:53:19.220256 16846 ts_tablet_manager.cc:1397] T 6e821a7d79e44edebb2b2be52fb03a22 P e6f4bc6c7c4d4b9583c043f619299d9c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 10:53:19.220709 16846 raft_consensus.cc:348] T 6e821a7d79e44edebb2b2be52fb03a22 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:19.220821 16846 raft_consensus.cc:374] T 6e821a7d79e44edebb2b2be52fb03a22 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:19.220860 16846 raft_consensus.cc:729] T 6e821a7d79e44edebb2b2be52fb03a22 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f4bc6c7c4d4b9583c043f619299d9c, State: Initialized, Role: FOLLOWER
I20260927 10:53:19.220989 16846 consensus_queue.cc:260] T 6e821a7d79e44edebb2b2be52fb03a22 P e6f4bc6c7c4d4b9583c043f619299d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:19.221276 16846 ts_tablet_manager.cc:1428] T 6e821a7d79e44edebb2b2be52fb03a22 P e6f4bc6c7c4d4b9583c043f619299d9c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:53:19.221698 16853 tablet_replica.cc:439] T 6e821a7d79e44edebb2b2be52fb03a22 P e6f4bc6c7c4d4b9583c043f619299d9c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } } }
I20260927 10:53:19.221870 16853 tablet_replica.cc:442] T 6e821a7d79e44edebb2b2be52fb03a22 P e6f4bc6c7c4d4b9583c043f619299d9c: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 10:53:19.230862 16543 tablet.cc:2369] T 6e821a7d79e44edebb2b2be52fb03a22 P 5cbbac417b1048d4b0139df810615709: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:53:19.503832 16852 raft_consensus.cc:482] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:19.504001 16852 raft_consensus.cc:504] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:19.505265 16852 leader_election.cc:288] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cbbac417b1048d4b0139df810615709 (127.14.68.1:45141), e6f4bc6c7c4d4b9583c043f619299d9c (127.14.68.2:37707)
I20260927 10:53:19.509092 16841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:19.505551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:56755 (local address 127.14.68.1:45141)
0927 10:53:19.505769 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:19.505773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:19.505887 (+ 114us) server_negotiation.cc:407] Connection header received
0927 10:53:19.506077 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:19.506082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:19.506134 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:19.506214 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:19.506479 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.506995 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.508152 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.508542 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.508669 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:19.508735 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:19.508905 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 10:53:19.508971 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260927 10:53:19.509158 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:19.505469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:45141 (local address 127.14.68.3:56755)
0927 10:53:19.505807 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:19.505824 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:19.505940 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:19.506235 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:19.506238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:19.506255 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:19.506377 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:19.506382 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.507111 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.507114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:19.508018 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:19.508026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.508668 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.508671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:19.508774 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:19.508795 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:53:19.508892 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 10:53:19.508963 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":21,"thread_start_us":96,"threads_started":1}
I20260927 10:53:19.509174 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:19.505723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:46729 (local address 127.14.68.2:37707)
0927 10:53:19.505961 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:19.505977 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:19.506020 (+ 43us) server_negotiation.cc:407] Connection header received
0927 10:53:19.506219 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:19.506224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:19.506270 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:19.506361 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:19.506611 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.507132 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.508220 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.508616 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.508741 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:19.508800 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:19.508974 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 10:53:19.509039 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":80,"threads_started":1}
I20260927 10:53:19.509552 16866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:19.505493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:37707 (local address 127.14.68.3:46729)
0927 10:53:19.505942 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:19.505956 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:19.506093 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:19.506373 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:19.506376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:19.506392 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:19.506509 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:19.506514 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.507241 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.507244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:19.508096 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:19.508104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.508741 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.508745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:19.508844 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:19.508864 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:53:19.508958 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 10:53:19.509025 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":15,"thread_start_us":73,"threads_started":1}
I20260927 10:53:19.509577 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e821a7d79e44edebb2b2be52fb03a22" candidate_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cbbac417b1048d4b0139df810615709" is_pre_election: true
I20260927 10:53:19.509718 16496 raft_consensus.cc:2454] T 6e821a7d79e44edebb2b2be52fb03a22 P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de1e8de15ff04a93a0cfdc98e9312d70 in term 0.
I20260927 10:53:19.510069 16709 leader_election.cc:302] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cbbac417b1048d4b0139df810615709, de1e8de15ff04a93a0cfdc98e9312d70; no voters:
I20260927 10:53:19.510131 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e821a7d79e44edebb2b2be52fb03a22" candidate_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" is_pre_election: true
I20260927 10:53:19.510334 16636 raft_consensus.cc:2454] T 6e821a7d79e44edebb2b2be52fb03a22 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de1e8de15ff04a93a0cfdc98e9312d70 in term 0.
I20260927 10:53:19.510370 16852 raft_consensus.cc:2781] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:19.510433 16852 raft_consensus.cc:482] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:19.510489 16852 raft_consensus.cc:3037] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:19.511565 16852 raft_consensus.cc:504] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:19.511919 16852 leader_election.cc:288] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70 [CANDIDATE]: Term 1 election: Requested vote from peers 5cbbac417b1048d4b0139df810615709 (127.14.68.1:45141), e6f4bc6c7c4d4b9583c043f619299d9c (127.14.68.2:37707)
I20260927 10:53:19.512254 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e821a7d79e44edebb2b2be52fb03a22" candidate_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cbbac417b1048d4b0139df810615709"
I20260927 10:53:19.512250 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e821a7d79e44edebb2b2be52fb03a22" candidate_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c"
I20260927 10:53:19.512360 16636 raft_consensus.cc:3037] T 6e821a7d79e44edebb2b2be52fb03a22 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:19.512375 16496 raft_consensus.cc:3037] T 6e821a7d79e44edebb2b2be52fb03a22 P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:19.513203 16496 raft_consensus.cc:2454] T 6e821a7d79e44edebb2b2be52fb03a22 P 5cbbac417b1048d4b0139df810615709 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de1e8de15ff04a93a0cfdc98e9312d70 in term 1.
I20260927 10:53:19.513223 16636 raft_consensus.cc:2454] T 6e821a7d79e44edebb2b2be52fb03a22 P e6f4bc6c7c4d4b9583c043f619299d9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de1e8de15ff04a93a0cfdc98e9312d70 in term 1.
I20260927 10:53:19.513533 16708 leader_election.cc:302] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de1e8de15ff04a93a0cfdc98e9312d70, e6f4bc6c7c4d4b9583c043f619299d9c; no voters:
I20260927 10:53:19.513731 16852 raft_consensus.cc:2781] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:19.513886 16852 raft_consensus.cc:686] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70 [term 1 LEADER]: Becoming Leader. State: Replica: de1e8de15ff04a93a0cfdc98e9312d70, State: Running, Role: LEADER
I20260927 10:53:19.514190 16852 consensus_queue.cc:237] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:19.516357 16869 tablet_replica.cc:439] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70: TxnStatusTablet state changed. Reason: New leader de1e8de15ff04a93a0cfdc98e9312d70. Latest consensus state: current_term: 1 leader_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } } }
I20260927 10:53:19.516472 16869 tablet_replica.cc:442] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:53:19.516723 16871 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 10:53:19.516917 16871 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 10:53:19.517603 16338 catalog_manager.cc:5495] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70 reported cstate change: term changed from 0 to 1, leader changed from <none> to de1e8de15ff04a93a0cfdc98e9312d70 (127.14.68.3). New cstate: current_term: 1 leader_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } health_report { overall_health: HEALTHY } } }
I20260927 10:53:19.529142 16853 consensus_queue.cc:1035] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:19.533565 16875 consensus_queue.cc:1035] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:19.585132 16843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:19.581428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57140 (local address 127.14.68.3:46341)
0927 10:53:19.581562 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:19.581567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:19.581583 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:53:19.581762 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:19.581765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:19.581820 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:19.581922 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:19.582162 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.582685 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.583625 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.583871 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.584010 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:19.584301 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:19.584721 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:19.584844 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:19.584933 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 10:53:19.585000 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:53:19.585131 16862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:19.581316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:46341 (local address 127.0.0.1:57140)
0927 10:53:19.581447 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:19.581465 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:19.581580 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:19.581934 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:19.581937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:19.581944 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:53:19.582053 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:19.582059 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.582795 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.582799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:19.583494 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:19.583504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.584006 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.584009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:19.584101 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:19.584154 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:53:19.584821 (+ 667us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:53:19.584838 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 10:53:19.584907 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 10:53:19.584992 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 10:53:19.590294 16496 raft_consensus.cc:1261] T 6e821a7d79e44edebb2b2be52fb03a22 P 5cbbac417b1048d4b0139df810615709 [term 1 FOLLOWER]: Refusing update from remote peer de1e8de15ff04a93a0cfdc98e9312d70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:19.590498 16636 raft_consensus.cc:1261] T 6e821a7d79e44edebb2b2be52fb03a22 P e6f4bc6c7c4d4b9583c043f619299d9c [term 1 FOLLOWER]: Refusing update from remote peer de1e8de15ff04a93a0cfdc98e9312d70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:19.590767 16869 consensus_queue.cc:1035] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:19.591048 16852 consensus_queue.cc:1035] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:19.594400 16875 tablet_replica.cc:439] T 6e821a7d79e44edebb2b2be52fb03a22 P e6f4bc6c7c4d4b9583c043f619299d9c: TxnStatusTablet state changed. Reason: New leader de1e8de15ff04a93a0cfdc98e9312d70. Latest consensus state: current_term: 1 leader_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } } }
I20260927 10:53:19.594530 16875 tablet_replica.cc:442] T 6e821a7d79e44edebb2b2be52fb03a22 P e6f4bc6c7c4d4b9583c043f619299d9c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:53:19.595376 16851 tablet_replica.cc:439] T 6e821a7d79e44edebb2b2be52fb03a22 P 5cbbac417b1048d4b0139df810615709: TxnStatusTablet state changed. Reason: New leader de1e8de15ff04a93a0cfdc98e9312d70. Latest consensus state: current_term: 1 leader_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } } }
I20260927 10:53:19.595489 16851 tablet_replica.cc:442] T 6e821a7d79e44edebb2b2be52fb03a22 P 5cbbac417b1048d4b0139df810615709: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:53:19.596726 16852 tablet_replica.cc:439] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } } }
I20260927 10:53:19.596824 16852 tablet_replica.cc:442] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:53:19.596909 16876 mvcc.cc:205] Tried to move back new op lower bound from 7333914212715819008 to 7333914212412510208. Current Snapshot: MvccSnapshot[applied={T|T < 7333914212715819008}]
I20260927 10:53:19.598061 16851 tablet_replica.cc:439] T 6e821a7d79e44edebb2b2be52fb03a22 P 5cbbac417b1048d4b0139df810615709: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } } }
I20260927 10:53:19.598187 16851 tablet_replica.cc:442] T 6e821a7d79e44edebb2b2be52fb03a22 P 5cbbac417b1048d4b0139df810615709: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:53:19.598101 16878 mvcc.cc:205] Tried to move back new op lower bound from 7333914212715819008 to 7333914212412510208. Current Snapshot: MvccSnapshot[applied={T|T < 7333914212715819008}]
I20260927 10:53:19.599653 16869 tablet_replica.cc:439] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } } }
I20260927 10:53:19.599774 16869 tablet_replica.cc:442] T 6e821a7d79e44edebb2b2be52fb03a22 P de1e8de15ff04a93a0cfdc98e9312d70: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:53:19.602034 16874 mvcc.cc:205] Tried to move back new op lower bound from 7333914212715819008 to 7333914212412510208. Current Snapshot: MvccSnapshot[applied={T|T < 7333914212715819008}]
I20260927 10:53:19.602279 16875 tablet_replica.cc:439] T 6e821a7d79e44edebb2b2be52fb03a22 P e6f4bc6c7c4d4b9583c043f619299d9c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } } }
I20260927 10:53:19.602394 16875 tablet_replica.cc:442] T 6e821a7d79e44edebb2b2be52fb03a22 P e6f4bc6c7c4d4b9583c043f619299d9c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:53:19.607645 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:19.603869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41106 (local address 127.14.68.2:37707)
0927 10:53:19.603976 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:19.603980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:19.603995 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:53:19.604076 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:19.604081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:19.604148 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:19.604249 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:19.604527 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.605063 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.606331 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.606541 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.606619 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:19.606906 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:19.607273 (+ 367us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:19.607354 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:19.607459 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 10:53:19.607504 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 10:53:19.617970 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:19.614333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:46341 (local address 127.14.68.2:52123)
0927 10:53:19.614609 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:19.614622 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:19.614704 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:19.614926 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:19.614928 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:19.614933 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:53:19.615018 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:19.615022 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.616011 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.616014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:19.616646 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:19.616654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.617271 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.617274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:19.617344 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:19.617381 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:53:19.617723 (+ 342us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:53:19.617728 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 10:53:19.617782 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 10:53:19.617838 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":103,"threads_started":1}
I20260927 10:53:19.618034 16843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:19.614439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:52123 (local address 127.14.68.3:46341)
0927 10:53:19.614570 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:19.614575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:19.614690 (+ 115us) server_negotiation.cc:407] Connection header received
0927 10:53:19.614781 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:19.614783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:19.614839 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:19.614926 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:19.615112 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.615879 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.616796 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.617151 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.617255 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:19.617548 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:19.617629 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:19.617713 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:19.617800 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 10:53:19.617868 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:53:19.645273 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:19.641701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:39713 (local address 127.14.68.2:37707)
0927 10:53:19.641841 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:19.641845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:19.642052 (+ 207us) server_negotiation.cc:407] Connection header received
0927 10:53:19.642183 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:19.642187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:19.642239 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:19.642321 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:19.642605 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.643115 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.644069 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.644394 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.644509 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:19.644825 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:19.644909 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:19.644993 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:19.645091 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 10:53:19.645136 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:53:19.645303 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:19.641667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:37707 (local address 127.14.68.3:39713)
0927 10:53:19.641955 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:19.641972 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:19.642089 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:19.642348 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:19.642350 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:19.642362 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:53:19.642504 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:19.642513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.643220 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.643226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:19.643953 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:19.643962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:19.644521 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:19.644524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:19.644617 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:19.644668 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:53:19.644994 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:53:19.645000 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:53:19.645071 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 10:53:19.645148 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":104,"threads_started":1}
I20260927 10:53:19.675477 16338 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:19.675670 16338 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:19.678282 16338 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa: Sending DeleteTablet for 3 replicas of tablet ddc7d13e0a2f4516880e5eb797ac79fc
I20260927 10:53:19.679008 16476 tablet_service.cc:1514] Processing DeleteTablet for tablet ddc7d13e0a2f4516880e5eb797ac79fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:19 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:56228
I20260927 10:53:19.679407 16616 tablet_service.cc:1514] Processing DeleteTablet for tablet ddc7d13e0a2f4516880e5eb797ac79fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:19 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:41102
I20260927 10:53:19.679783 16895 tablet_replica.cc:330] T ddc7d13e0a2f4516880e5eb797ac79fc P 5cbbac417b1048d4b0139df810615709: stopping tablet replica
I20260927 10:53:19.679806 16756 tablet_service.cc:1514] Processing DeleteTablet for tablet ddc7d13e0a2f4516880e5eb797ac79fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:19 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:57128
I20260927 10:53:19.680199 16895 raft_consensus.cc:2229] T ddc7d13e0a2f4516880e5eb797ac79fc P 5cbbac417b1048d4b0139df810615709 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:19.680466 16895 raft_consensus.cc:2258] T ddc7d13e0a2f4516880e5eb797ac79fc P 5cbbac417b1048d4b0139df810615709 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:19.680879 16338 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:53:19.681270 16338 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:19.682196 16895 ts_tablet_manager.cc:1905] T ddc7d13e0a2f4516880e5eb797ac79fc P 5cbbac417b1048d4b0139df810615709: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:19.684536 16896 tablet_replica.cc:330] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c: stopping tablet replica
I20260927 10:53:19.684978 16897 tablet_replica.cc:330] T ddc7d13e0a2f4516880e5eb797ac79fc P de1e8de15ff04a93a0cfdc98e9312d70: stopping tablet replica
I20260927 10:53:19.685276 16897 raft_consensus.cc:2229] T ddc7d13e0a2f4516880e5eb797ac79fc P de1e8de15ff04a93a0cfdc98e9312d70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:19.685604 16895 ts_tablet_manager.cc:1918] T ddc7d13e0a2f4516880e5eb797ac79fc P 5cbbac417b1048d4b0139df810615709: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:53:19.685930 16476 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb08186b1ca485497648e09e2e9c2a3 (DEFAULT_TABLE table=test-table [id=e093978ae4164106af79c4678653b854]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:19.686183 16476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb08186b1ca485497648e09e2e9c2a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:19.686369 16756 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb08186b1ca485497648e09e2e9c2a3 (DEFAULT_TABLE table=test-table [id=e093978ae4164106af79c4678653b854]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:19.686370 16616 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb08186b1ca485497648e09e2e9c2a3 (DEFAULT_TABLE table=test-table [id=e093978ae4164106af79c4678653b854]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:19.686604 16616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb08186b1ca485497648e09e2e9c2a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:19.686604 16756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb08186b1ca485497648e09e2e9c2a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:19.688325 16845 tablet_bootstrap.cc:492] T 9cb08186b1ca485497648e09e2e9c2a3 P 5cbbac417b1048d4b0139df810615709: Bootstrap starting.
I20260927 10:53:19.688911 16847 tablet_bootstrap.cc:492] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70: Bootstrap starting.
I20260927 10:53:19.689044 16846 tablet_bootstrap.cc:492] T 9cb08186b1ca485497648e09e2e9c2a3 P e6f4bc6c7c4d4b9583c043f619299d9c: Bootstrap starting.
I20260927 10:53:19.689222 16845 tablet_bootstrap.cc:654] T 9cb08186b1ca485497648e09e2e9c2a3 P 5cbbac417b1048d4b0139df810615709: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:19.689679 16847 tablet_bootstrap.cc:654] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:19.689937 16846 tablet_bootstrap.cc:654] T 9cb08186b1ca485497648e09e2e9c2a3 P e6f4bc6c7c4d4b9583c043f619299d9c: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:19.690032 16895 log.cc:1196] T ddc7d13e0a2f4516880e5eb797ac79fc P 5cbbac417b1048d4b0139df810615709: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-0/wal/wals/ddc7d13e0a2f4516880e5eb797ac79fc
I20260927 10:53:19.690358 16895 ts_tablet_manager.cc:1939] T ddc7d13e0a2f4516880e5eb797ac79fc P 5cbbac417b1048d4b0139df810615709: Deleting consensus metadata
I20260927 10:53:19.691475 16325 catalog_manager.cc:4841] TS 5cbbac417b1048d4b0139df810615709 (127.14.68.1:45141): tablet ddc7d13e0a2f4516880e5eb797ac79fc (table test-table [id=7aea5ab12c564219b0eeb96745809305]) successfully deleted
I20260927 10:53:19.691612 16846 tablet_bootstrap.cc:492] T 9cb08186b1ca485497648e09e2e9c2a3 P e6f4bc6c7c4d4b9583c043f619299d9c: No bootstrap required, opened a new log
I20260927 10:53:19.691689 16846 ts_tablet_manager.cc:1397] T 9cb08186b1ca485497648e09e2e9c2a3 P e6f4bc6c7c4d4b9583c043f619299d9c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 10:53:19.692060 16846 raft_consensus.cc:348] T 9cb08186b1ca485497648e09e2e9c2a3 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:19.692116 16845 tablet_bootstrap.cc:492] T 9cb08186b1ca485497648e09e2e9c2a3 P 5cbbac417b1048d4b0139df810615709: No bootstrap required, opened a new log
I20260927 10:53:19.692188 16845 ts_tablet_manager.cc:1397] T 9cb08186b1ca485497648e09e2e9c2a3 P 5cbbac417b1048d4b0139df810615709: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 10:53:19.692181 16846 raft_consensus.cc:374] T 9cb08186b1ca485497648e09e2e9c2a3 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:19.692226 16846 raft_consensus.cc:729] T 9cb08186b1ca485497648e09e2e9c2a3 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f4bc6c7c4d4b9583c043f619299d9c, State: Initialized, Role: FOLLOWER
I20260927 10:53:19.692344 16846 consensus_queue.cc:260] T 9cb08186b1ca485497648e09e2e9c2a3 P e6f4bc6c7c4d4b9583c043f619299d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:19.692600 16846 ts_tablet_manager.cc:1428] T 9cb08186b1ca485497648e09e2e9c2a3 P e6f4bc6c7c4d4b9583c043f619299d9c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:53:19.692548 16845 raft_consensus.cc:348] T 9cb08186b1ca485497648e09e2e9c2a3 P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:19.692656 16845 raft_consensus.cc:374] T 9cb08186b1ca485497648e09e2e9c2a3 P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:19.692692 16845 raft_consensus.cc:729] T 9cb08186b1ca485497648e09e2e9c2a3 P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cbbac417b1048d4b0139df810615709, State: Initialized, Role: FOLLOWER
I20260927 10:53:19.692804 16845 consensus_queue.cc:260] T 9cb08186b1ca485497648e09e2e9c2a3 P 5cbbac417b1048d4b0139df810615709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:19.693025 16845 ts_tablet_manager.cc:1428] T 9cb08186b1ca485497648e09e2e9c2a3 P 5cbbac417b1048d4b0139df810615709: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:53:19.694183 16896 raft_consensus.cc:2229] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:53:19.685443 16897 raft_consensus.cc:2258] T ddc7d13e0a2f4516880e5eb797ac79fc P de1e8de15ff04a93a0cfdc98e9312d70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:19.695168 16896 raft_consensus.cc:2258] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:19.696573 16897 ts_tablet_manager.cc:1905] T ddc7d13e0a2f4516880e5eb797ac79fc P de1e8de15ff04a93a0cfdc98e9312d70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:19.698895 16897 ts_tablet_manager.cc:1918] T ddc7d13e0a2f4516880e5eb797ac79fc P de1e8de15ff04a93a0cfdc98e9312d70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:53:19.698990 16897 log.cc:1196] T ddc7d13e0a2f4516880e5eb797ac79fc P de1e8de15ff04a93a0cfdc98e9312d70: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/ddc7d13e0a2f4516880e5eb797ac79fc
I20260927 10:53:19.699339 16897 ts_tablet_manager.cc:1939] T ddc7d13e0a2f4516880e5eb797ac79fc P de1e8de15ff04a93a0cfdc98e9312d70: Deleting consensus metadata
I20260927 10:53:19.700363 16324 catalog_manager.cc:4841] TS de1e8de15ff04a93a0cfdc98e9312d70 (127.14.68.3:46341): tablet ddc7d13e0a2f4516880e5eb797ac79fc (table test-table [id=7aea5ab12c564219b0eeb96745809305]) successfully deleted
I20260927 10:53:19.700502 16896 ts_tablet_manager.cc:1905] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:19.703199 16896 ts_tablet_manager.cc:1918] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:53:19.703284 16896 log.cc:1196] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/ddc7d13e0a2f4516880e5eb797ac79fc
I20260927 10:53:19.703553 16896 ts_tablet_manager.cc:1939] T ddc7d13e0a2f4516880e5eb797ac79fc P e6f4bc6c7c4d4b9583c043f619299d9c: Deleting consensus metadata
I20260927 10:53:19.704321 16847 tablet_bootstrap.cc:492] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70: No bootstrap required, opened a new log
I20260927 10:53:19.704424 16847 ts_tablet_manager.cc:1397] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.001s
I20260927 10:53:19.704609 16324 catalog_manager.cc:4841] TS e6f4bc6c7c4d4b9583c043f619299d9c (127.14.68.2:37707): tablet ddc7d13e0a2f4516880e5eb797ac79fc (table test-table [id=7aea5ab12c564219b0eeb96745809305]) successfully deleted
I20260927 10:53:19.704874 16847 raft_consensus.cc:348] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:19.704989 16847 raft_consensus.cc:374] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:19.705031 16847 raft_consensus.cc:729] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de1e8de15ff04a93a0cfdc98e9312d70, State: Initialized, Role: FOLLOWER
I20260927 10:53:19.705163 16847 consensus_queue.cc:260] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:19.705463 16847 ts_tablet_manager.cc:1428] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:53:20.041364 16890 raft_consensus.cc:482] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:20.041504 16890 raft_consensus.cc:504] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:20.041894 16890 leader_election.cc:288] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cbbac417b1048d4b0139df810615709 (127.14.68.1:45141), e6f4bc6c7c4d4b9583c043f619299d9c (127.14.68.2:37707)
I20260927 10:53:20.042416 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb08186b1ca485497648e09e2e9c2a3" candidate_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cbbac417b1048d4b0139df810615709" is_pre_election: true
I20260927 10:53:20.042436 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb08186b1ca485497648e09e2e9c2a3" candidate_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" is_pre_election: true
I20260927 10:53:20.042610 16636 raft_consensus.cc:2454] T 9cb08186b1ca485497648e09e2e9c2a3 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de1e8de15ff04a93a0cfdc98e9312d70 in term 0.
I20260927 10:53:20.042608 16496 raft_consensus.cc:2454] T 9cb08186b1ca485497648e09e2e9c2a3 P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de1e8de15ff04a93a0cfdc98e9312d70 in term 0.
I20260927 10:53:20.042949 16708 leader_election.cc:302] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5cbbac417b1048d4b0139df810615709, de1e8de15ff04a93a0cfdc98e9312d70, e6f4bc6c7c4d4b9583c043f619299d9c; no voters:
I20260927 10:53:20.043129 16890 raft_consensus.cc:2781] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:20.043187 16890 raft_consensus.cc:482] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:20.043215 16890 raft_consensus.cc:3037] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:20.044289 16890 raft_consensus.cc:504] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:20.044637 16890 leader_election.cc:288] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70 [CANDIDATE]: Term 1 election: Requested vote from peers 5cbbac417b1048d4b0139df810615709 (127.14.68.1:45141), e6f4bc6c7c4d4b9583c043f619299d9c (127.14.68.2:37707)
I20260927 10:53:20.044986 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb08186b1ca485497648e09e2e9c2a3" candidate_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cbbac417b1048d4b0139df810615709"
I20260927 10:53:20.045114 16496 raft_consensus.cc:3037] T 9cb08186b1ca485497648e09e2e9c2a3 P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:20.045120 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb08186b1ca485497648e09e2e9c2a3" candidate_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c"
I20260927 10:53:20.045274 16636 raft_consensus.cc:3037] T 9cb08186b1ca485497648e09e2e9c2a3 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:20.046036 16496 raft_consensus.cc:2454] T 9cb08186b1ca485497648e09e2e9c2a3 P 5cbbac417b1048d4b0139df810615709 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de1e8de15ff04a93a0cfdc98e9312d70 in term 1.
I20260927 10:53:20.046283 16636 raft_consensus.cc:2454] T 9cb08186b1ca485497648e09e2e9c2a3 P e6f4bc6c7c4d4b9583c043f619299d9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de1e8de15ff04a93a0cfdc98e9312d70 in term 1.
I20260927 10:53:20.046327 16709 leader_election.cc:302] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cbbac417b1048d4b0139df810615709, de1e8de15ff04a93a0cfdc98e9312d70; no voters:
I20260927 10:53:20.046504 16890 raft_consensus.cc:2781] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:20.046609 16890 raft_consensus.cc:686] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70 [term 1 LEADER]: Becoming Leader. State: Replica: de1e8de15ff04a93a0cfdc98e9312d70, State: Running, Role: LEADER
I20260927 10:53:20.046749 16890 consensus_queue.cc:237] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:20.048359 16337 catalog_manager.cc:5495] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70 reported cstate change: term changed from 0 to 1, leader changed from <none> to de1e8de15ff04a93a0cfdc98e9312d70 (127.14.68.3). New cstate: current_term: 1 leader_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } health_report { overall_health: HEALTHY } } }
I20260927 10:53:20.066232 16843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:20.063066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57142 (local address 127.14.68.3:46341)
0927 10:53:20.063228 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:20.063233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:20.063246 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:53:20.063376 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:20.063380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:20.063424 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:20.063510 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:20.063790 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:20.064323 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:20.065154 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:20.065383 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:20.065492 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:20.065748 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:20.065828 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:20.065903 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:20.066025 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 10:53:20.066086 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:53:20.074131 16843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:20.071625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:50277 (local address 127.14.68.3:46341)
0927 10:53:20.071774 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:20.071777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:20.071789 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:53:20.071868 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:20.071870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:20.071906 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:20.071966 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:20.072139 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:20.072591 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:20.073351 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:20.073560 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:20.073637 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:20.073815 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:20.073872 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:20.073921 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:20.073973 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 10:53:20.074008 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 10:53:20.074154 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:20.071534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:46341 (local address 127.14.68.3:50277)
0927 10:53:20.071664 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:20.071685 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:20.071795 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:20.071981 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:20.071982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:20.071987 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:53:20.072057 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:20.072061 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:20.072688 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:20.072690 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:20.073259 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:20.073266 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:20.073642 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:20.073645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:20.073702 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:20.073725 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:53:20.073923 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:53:20.073926 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 10:53:20.073966 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 10:53:20.074025 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 10:53:20.079538 16496 raft_consensus.cc:1261] T 9cb08186b1ca485497648e09e2e9c2a3 P 5cbbac417b1048d4b0139df810615709 [term 1 FOLLOWER]: Refusing update from remote peer de1e8de15ff04a93a0cfdc98e9312d70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:20.079656 16636 raft_consensus.cc:1261] T 9cb08186b1ca485497648e09e2e9c2a3 P e6f4bc6c7c4d4b9583c043f619299d9c [term 1 FOLLOWER]: Refusing update from remote peer de1e8de15ff04a93a0cfdc98e9312d70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:20.079907 16869 consensus_queue.cc:1035] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:20.080036 16852 consensus_queue.cc:1035] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 10:53:20.093451 16876 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 10:53:21.645709 16339 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:21.645851 16339 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:21.648075 16339 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa: Sending DeleteTablet for 3 replicas of tablet 9cb08186b1ca485497648e09e2e9c2a3
I20260927 10:53:21.648586 16755 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cb08186b1ca485497648e09e2e9c2a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:21 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:57128
I20260927 10:53:21.648586 16476 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cb08186b1ca485497648e09e2e9c2a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:21 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:56228
I20260927 10:53:21.648893 16616 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cb08186b1ca485497648e09e2e9c2a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:21 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:41102
I20260927 10:53:21.649260 16927 tablet_replica.cc:330] T 9cb08186b1ca485497648e09e2e9c2a3 P e6f4bc6c7c4d4b9583c043f619299d9c: stopping tablet replica
I20260927 10:53:21.649426 16927 raft_consensus.cc:2229] T 9cb08186b1ca485497648e09e2e9c2a3 P e6f4bc6c7c4d4b9583c043f619299d9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:21.649557 16927 raft_consensus.cc:2258] T 9cb08186b1ca485497648e09e2e9c2a3 P e6f4bc6c7c4d4b9583c043f619299d9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:21.649804 16926 tablet_replica.cc:330] T 9cb08186b1ca485497648e09e2e9c2a3 P 5cbbac417b1048d4b0139df810615709: stopping tablet replica
I20260927 10:53:21.649925 16926 raft_consensus.cc:2229] T 9cb08186b1ca485497648e09e2e9c2a3 P 5cbbac417b1048d4b0139df810615709 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:21.650051 16926 raft_consensus.cc:2258] T 9cb08186b1ca485497648e09e2e9c2a3 P 5cbbac417b1048d4b0139df810615709 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:21.650537 16927 ts_tablet_manager.cc:1905] T 9cb08186b1ca485497648e09e2e9c2a3 P e6f4bc6c7c4d4b9583c043f619299d9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:21.652684 16927 ts_tablet_manager.cc:1918] T 9cb08186b1ca485497648e09e2e9c2a3 P e6f4bc6c7c4d4b9583c043f619299d9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:53:21.652798 16927 log.cc:1196] T 9cb08186b1ca485497648e09e2e9c2a3 P e6f4bc6c7c4d4b9583c043f619299d9c: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/9cb08186b1ca485497648e09e2e9c2a3
I20260927 10:53:21.653076 16927 ts_tablet_manager.cc:1939] T 9cb08186b1ca485497648e09e2e9c2a3 P e6f4bc6c7c4d4b9583c043f619299d9c: Deleting consensus metadata
I20260927 10:53:21.653556 16926 ts_tablet_manager.cc:1905] T 9cb08186b1ca485497648e09e2e9c2a3 P 5cbbac417b1048d4b0139df810615709: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:21.653884 16324 catalog_manager.cc:4841] TS e6f4bc6c7c4d4b9583c043f619299d9c (127.14.68.2:37707): tablet 9cb08186b1ca485497648e09e2e9c2a3 (table test-table [id=e093978ae4164106af79c4678653b854]) successfully deleted
I20260927 10:53:21.655889 16925 tablet_replica.cc:330] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70: stopping tablet replica
I20260927 10:53:21.656046 16926 ts_tablet_manager.cc:1918] T 9cb08186b1ca485497648e09e2e9c2a3 P 5cbbac417b1048d4b0139df810615709: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:53:21.656093 16925 raft_consensus.cc:2229] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:53:21.656129 16926 log.cc:1196] T 9cb08186b1ca485497648e09e2e9c2a3 P 5cbbac417b1048d4b0139df810615709: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-0/wal/wals/9cb08186b1ca485497648e09e2e9c2a3
I20260927 10:53:21.656395 16926 ts_tablet_manager.cc:1939] T 9cb08186b1ca485497648e09e2e9c2a3 P 5cbbac417b1048d4b0139df810615709: Deleting consensus metadata
I20260927 10:53:21.656399 16925 raft_consensus.cc:2258] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:21.657053 16925 ts_tablet_manager.cc:1905] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:21.657091 16325 catalog_manager.cc:4841] TS 5cbbac417b1048d4b0139df810615709 (127.14.68.1:45141): tablet 9cb08186b1ca485497648e09e2e9c2a3 (table test-table [id=e093978ae4164106af79c4678653b854]) successfully deleted
I20260927 10:53:21.659000 16925 ts_tablet_manager.cc:1918] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:53:21.659096 16925 log.cc:1196] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/9cb08186b1ca485497648e09e2e9c2a3
I20260927 10:53:21.659354 16925 ts_tablet_manager.cc:1939] T 9cb08186b1ca485497648e09e2e9c2a3 P de1e8de15ff04a93a0cfdc98e9312d70: Deleting consensus metadata
I20260927 10:53:21.660012 16324 catalog_manager.cc:4841] TS de1e8de15ff04a93a0cfdc98e9312d70 (127.14.68.3:46341): tablet 9cb08186b1ca485497648e09e2e9c2a3 (table test-table [id=e093978ae4164106af79c4678653b854]) successfully deleted
Sep 27 10:53:21 dist-test-slave-tz51 krb5kdc[16294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506401, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:53:21 dist-test-slave-tz51 krb5kdc[16294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506401, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:21.683902 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:21.668872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33244 (local address 127.14.68.62:39029)
0927 10:53:21.669161 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:21.669171 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:21.669254 (+ 83us) server_negotiation.cc:407] Connection header received
0927 10:53:21.669448 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:21.669451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:21.669494 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:21.669606 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:21.670325 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:21.670884 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:21.671782 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:21.672015 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:21.672134 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:21.680587 (+ 8453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:21.680591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:21.680593 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:21.680621 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:21.682162 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:21.682618 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:21.682620 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:21.682622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:21.682701 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:21.683003 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:21.683007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:21.683009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:21.683203 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:21.683309 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:21.683565 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 10:53:21.683763 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":105,"threads_started":1}
I20260927 10:53:21.686587 16339 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:53:21.687021 16339 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:21.691571 16476 tablet_service.cc:1467] Processing CreateTablet for tablet 0e4b21981317447b88cf412fe371bc84 (DEFAULT_TABLE table=test-table [id=95b29c86731d4527a42f8643b3504a31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:21.691928 16476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e4b21981317447b88cf412fe371bc84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:21.691884 16755 tablet_service.cc:1467] Processing CreateTablet for tablet 0e4b21981317447b88cf412fe371bc84 (DEFAULT_TABLE table=test-table [id=95b29c86731d4527a42f8643b3504a31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:21.692150 16755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e4b21981317447b88cf412fe371bc84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:21.694037 16936 tablet_bootstrap.cc:492] T 0e4b21981317447b88cf412fe371bc84 P 5cbbac417b1048d4b0139df810615709: Bootstrap starting.
I20260927 10:53:21.694285 16616 tablet_service.cc:1467] Processing CreateTablet for tablet 0e4b21981317447b88cf412fe371bc84 (DEFAULT_TABLE table=test-table [id=95b29c86731d4527a42f8643b3504a31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:21.694586 16616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e4b21981317447b88cf412fe371bc84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:21.695147 16936 tablet_bootstrap.cc:654] T 0e4b21981317447b88cf412fe371bc84 P 5cbbac417b1048d4b0139df810615709: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:21.697232 16939 tablet_bootstrap.cc:492] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c: Bootstrap starting.
I20260927 10:53:21.697861 16937 tablet_bootstrap.cc:492] T 0e4b21981317447b88cf412fe371bc84 P de1e8de15ff04a93a0cfdc98e9312d70: Bootstrap starting.
I20260927 10:53:21.698001 16939 tablet_bootstrap.cc:654] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:21.698202 16936 tablet_bootstrap.cc:492] T 0e4b21981317447b88cf412fe371bc84 P 5cbbac417b1048d4b0139df810615709: No bootstrap required, opened a new log
I20260927 10:53:21.698335 16936 ts_tablet_manager.cc:1397] T 0e4b21981317447b88cf412fe371bc84 P 5cbbac417b1048d4b0139df810615709: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 10:53:21.698729 16937 tablet_bootstrap.cc:654] T 0e4b21981317447b88cf412fe371bc84 P de1e8de15ff04a93a0cfdc98e9312d70: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:21.698846 16936 raft_consensus.cc:348] T 0e4b21981317447b88cf412fe371bc84 P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:21.698972 16936 raft_consensus.cc:374] T 0e4b21981317447b88cf412fe371bc84 P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:21.699023 16936 raft_consensus.cc:729] T 0e4b21981317447b88cf412fe371bc84 P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cbbac417b1048d4b0139df810615709, State: Initialized, Role: FOLLOWER
I20260927 10:53:21.699168 16936 consensus_queue.cc:260] T 0e4b21981317447b88cf412fe371bc84 P 5cbbac417b1048d4b0139df810615709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:21.699913 16939 tablet_bootstrap.cc:492] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c: No bootstrap required, opened a new log
I20260927 10:53:21.699987 16939 ts_tablet_manager.cc:1397] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:53:21.700179 16936 ts_tablet_manager.cc:1428] T 0e4b21981317447b88cf412fe371bc84 P 5cbbac417b1048d4b0139df810615709: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:53:21.700332 16939 raft_consensus.cc:348] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:21.700423 16939 raft_consensus.cc:374] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:21.700449 16939 raft_consensus.cc:729] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f4bc6c7c4d4b9583c043f619299d9c, State: Initialized, Role: FOLLOWER
I20260927 10:53:21.700555 16939 consensus_queue.cc:260] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:21.700882 16939 ts_tablet_manager.cc:1428] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:53:21.701527 16937 tablet_bootstrap.cc:492] T 0e4b21981317447b88cf412fe371bc84 P de1e8de15ff04a93a0cfdc98e9312d70: No bootstrap required, opened a new log
I20260927 10:53:21.701633 16937 ts_tablet_manager.cc:1397] T 0e4b21981317447b88cf412fe371bc84 P de1e8de15ff04a93a0cfdc98e9312d70: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 10:53:21.701939 16937 raft_consensus.cc:348] T 0e4b21981317447b88cf412fe371bc84 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:21.702011 16937 raft_consensus.cc:374] T 0e4b21981317447b88cf412fe371bc84 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:21.702030 16937 raft_consensus.cc:729] T 0e4b21981317447b88cf412fe371bc84 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de1e8de15ff04a93a0cfdc98e9312d70, State: Initialized, Role: FOLLOWER
I20260927 10:53:21.702158 16937 consensus_queue.cc:260] T 0e4b21981317447b88cf412fe371bc84 P de1e8de15ff04a93a0cfdc98e9312d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:21.702428 16937 ts_tablet_manager.cc:1428] T 0e4b21981317447b88cf412fe371bc84 P de1e8de15ff04a93a0cfdc98e9312d70: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:53:21.704834 16943 raft_consensus.cc:482] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:21.704936 16943 raft_consensus.cc:504] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:21.705358 16943 leader_election.cc:288] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cbbac417b1048d4b0139df810615709 (127.14.68.1:45141), de1e8de15ff04a93a0cfdc98e9312d70 (127.14.68.3:46341)
I20260927 10:53:21.705708 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4b21981317447b88cf412fe371bc84" candidate_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cbbac417b1048d4b0139df810615709" is_pre_election: true
I20260927 10:53:21.705869 16496 raft_consensus.cc:2454] T 0e4b21981317447b88cf412fe371bc84 P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6f4bc6c7c4d4b9583c043f619299d9c in term 0.
I20260927 10:53:21.706178 16569 leader_election.cc:302] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cbbac417b1048d4b0139df810615709, e6f4bc6c7c4d4b9583c043f619299d9c; no voters:
I20260927 10:53:21.706337 16943 raft_consensus.cc:2781] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:21.706398 16943 raft_consensus.cc:482] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:21.706357 16776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4b21981317447b88cf412fe371bc84" candidate_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" is_pre_election: true
I20260927 10:53:21.706441 16943 raft_consensus.cc:3037] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:21.706475 16776 raft_consensus.cc:2454] T 0e4b21981317447b88cf412fe371bc84 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6f4bc6c7c4d4b9583c043f619299d9c in term 0.
I20260927 10:53:21.707363 16943 raft_consensus.cc:504] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:21.707702 16943 leader_election.cc:288] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c [CANDIDATE]: Term 1 election: Requested vote from peers 5cbbac417b1048d4b0139df810615709 (127.14.68.1:45141), de1e8de15ff04a93a0cfdc98e9312d70 (127.14.68.3:46341)
I20260927 10:53:21.708045 16776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4b21981317447b88cf412fe371bc84" candidate_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de1e8de15ff04a93a0cfdc98e9312d70"
I20260927 10:53:21.708150 16776 raft_consensus.cc:3037] T 0e4b21981317447b88cf412fe371bc84 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:21.708967 16776 raft_consensus.cc:2454] T 0e4b21981317447b88cf412fe371bc84 P de1e8de15ff04a93a0cfdc98e9312d70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6f4bc6c7c4d4b9583c043f619299d9c in term 1.
I20260927 10:53:21.709197 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4b21981317447b88cf412fe371bc84" candidate_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cbbac417b1048d4b0139df810615709"
I20260927 10:53:21.709256 16568 leader_election.cc:302] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de1e8de15ff04a93a0cfdc98e9312d70, e6f4bc6c7c4d4b9583c043f619299d9c; no voters:
I20260927 10:53:21.709311 16496 raft_consensus.cc:3037] T 0e4b21981317447b88cf412fe371bc84 P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:21.709448 16943 raft_consensus.cc:2781] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:21.709646 16943 raft_consensus.cc:686] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c [term 1 LEADER]: Becoming Leader. State: Replica: e6f4bc6c7c4d4b9583c043f619299d9c, State: Running, Role: LEADER
I20260927 10:53:21.709775 16943 consensus_queue.cc:237] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:21.710386 16496 raft_consensus.cc:2454] T 0e4b21981317447b88cf412fe371bc84 P 5cbbac417b1048d4b0139df810615709 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6f4bc6c7c4d4b9583c043f619299d9c in term 1.
I20260927 10:53:21.711275 16339 catalog_manager.cc:5495] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c reported cstate change: term changed from 0 to 1, leader changed from <none> to e6f4bc6c7c4d4b9583c043f619299d9c (127.14.68.2). New cstate: current_term: 1 leader_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } health_report { overall_health: UNKNOWN } } }
I20260927 10:53:21.730124 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:21.727089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41110 (local address 127.14.68.2:37707)
0927 10:53:21.727365 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:21.727372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:21.727384 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:53:21.727421 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:21.727423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:21.727464 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:21.727534 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:21.727823 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:21.728363 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:21.729175 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:21.729387 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:21.729459 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:21.729669 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:21.729733 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:21.729785 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:21.729863 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 10:53:21.729904 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":140,"threads_started":1}
I20260927 10:53:21.736035 16496 raft_consensus.cc:1261] T 0e4b21981317447b88cf412fe371bc84 P 5cbbac417b1048d4b0139df810615709 [term 1 FOLLOWER]: Refusing update from remote peer e6f4bc6c7c4d4b9583c043f619299d9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:21.736035 16776 raft_consensus.cc:1261] T 0e4b21981317447b88cf412fe371bc84 P de1e8de15ff04a93a0cfdc98e9312d70 [term 1 FOLLOWER]: Refusing update from remote peer e6f4bc6c7c4d4b9583c043f619299d9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:21.736472 16943 consensus_queue.cc:1035] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:21.736645 16944 consensus_queue.cc:1035] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:21.766978 16338 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:21.767148 16338 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:21.768929 16338 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa: Sending DeleteTablet for 3 replicas of tablet 0e4b21981317447b88cf412fe371bc84
I20260927 10:53:21.769418 16476 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e4b21981317447b88cf412fe371bc84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:21 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:56228
I20260927 10:53:21.769451 16616 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e4b21981317447b88cf412fe371bc84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:21 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:41102
I20260927 10:53:21.769629 16926 tablet_replica.cc:330] T 0e4b21981317447b88cf412fe371bc84 P 5cbbac417b1048d4b0139df810615709: stopping tablet replica
I20260927 10:53:21.769631 16927 tablet_replica.cc:330] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c: stopping tablet replica
I20260927 10:53:21.769623 16755 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e4b21981317447b88cf412fe371bc84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:21 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:57128
I20260927 10:53:21.769754 16927 raft_consensus.cc:2229] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:53:21.769754 16926 raft_consensus.cc:2229] T 0e4b21981317447b88cf412fe371bc84 P 5cbbac417b1048d4b0139df810615709 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:21.769797 16925 tablet_replica.cc:330] T 0e4b21981317447b88cf412fe371bc84 P de1e8de15ff04a93a0cfdc98e9312d70: stopping tablet replica
I20260927 10:53:21.769852 16926 raft_consensus.cc:2258] T 0e4b21981317447b88cf412fe371bc84 P 5cbbac417b1048d4b0139df810615709 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:21.769896 16925 raft_consensus.cc:2229] T 0e4b21981317447b88cf412fe371bc84 P de1e8de15ff04a93a0cfdc98e9312d70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:21.769923 16927 raft_consensus.cc:2258] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:21.770005 16925 raft_consensus.cc:2258] T 0e4b21981317447b88cf412fe371bc84 P de1e8de15ff04a93a0cfdc98e9312d70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:21.770414 16338 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:53:21.770814 16338 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:21.770843 16927 ts_tablet_manager.cc:1905] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:21.772133 16925 ts_tablet_manager.cc:1905] T 0e4b21981317447b88cf412fe371bc84 P de1e8de15ff04a93a0cfdc98e9312d70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:21.773303 16927 ts_tablet_manager.cc:1918] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:53:21.773381 16927 log.cc:1196] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/0e4b21981317447b88cf412fe371bc84
I20260927 10:53:21.773658 16927 ts_tablet_manager.cc:1939] T 0e4b21981317447b88cf412fe371bc84 P e6f4bc6c7c4d4b9583c043f619299d9c: Deleting consensus metadata
I20260927 10:53:21.774358 16925 ts_tablet_manager.cc:1918] T 0e4b21981317447b88cf412fe371bc84 P de1e8de15ff04a93a0cfdc98e9312d70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:53:21.774442 16925 log.cc:1196] T 0e4b21981317447b88cf412fe371bc84 P de1e8de15ff04a93a0cfdc98e9312d70: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/0e4b21981317447b88cf412fe371bc84
I20260927 10:53:21.774720 16925 ts_tablet_manager.cc:1939] T 0e4b21981317447b88cf412fe371bc84 P de1e8de15ff04a93a0cfdc98e9312d70: Deleting consensus metadata
I20260927 10:53:21.775404 16324 catalog_manager.cc:4841] TS de1e8de15ff04a93a0cfdc98e9312d70 (127.14.68.3:46341): tablet 0e4b21981317447b88cf412fe371bc84 (table test-table [id=95b29c86731d4527a42f8643b3504a31]) successfully deleted
I20260927 10:53:21.775549 16324 catalog_manager.cc:4841] TS e6f4bc6c7c4d4b9583c043f619299d9c (127.14.68.2:37707): tablet 0e4b21981317447b88cf412fe371bc84 (table test-table [id=95b29c86731d4527a42f8643b3504a31]) successfully deleted
I20260927 10:53:21.775593 16476 tablet_service.cc:1467] Processing CreateTablet for tablet 35506fbcf5994a0d86ee362df99305e6 (DEFAULT_TABLE table=test-table [id=9af54c98c9be4ea0b0016b7cc1aa76cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:21.775895 16476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35506fbcf5994a0d86ee362df99305e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:21.775998 16616 tablet_service.cc:1467] Processing CreateTablet for tablet 35506fbcf5994a0d86ee362df99305e6 (DEFAULT_TABLE table=test-table [id=9af54c98c9be4ea0b0016b7cc1aa76cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:21.776033 16755 tablet_service.cc:1467] Processing CreateTablet for tablet 35506fbcf5994a0d86ee362df99305e6 (DEFAULT_TABLE table=test-table [id=9af54c98c9be4ea0b0016b7cc1aa76cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:21.776177 16616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35506fbcf5994a0d86ee362df99305e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:21.776259 16755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35506fbcf5994a0d86ee362df99305e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:21.777802 16939 tablet_bootstrap.cc:492] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c: Bootstrap starting.
I20260927 10:53:21.778234 16936 tablet_bootstrap.cc:492] T 35506fbcf5994a0d86ee362df99305e6 P 5cbbac417b1048d4b0139df810615709: Bootstrap starting.
I20260927 10:53:21.778414 16937 tablet_bootstrap.cc:492] T 35506fbcf5994a0d86ee362df99305e6 P de1e8de15ff04a93a0cfdc98e9312d70: Bootstrap starting.
I20260927 10:53:21.778467 16926 ts_tablet_manager.cc:1905] T 0e4b21981317447b88cf412fe371bc84 P 5cbbac417b1048d4b0139df810615709: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:21.778674 16939 tablet_bootstrap.cc:654] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:21.779116 16936 tablet_bootstrap.cc:654] T 35506fbcf5994a0d86ee362df99305e6 P 5cbbac417b1048d4b0139df810615709: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:21.779120 16937 tablet_bootstrap.cc:654] T 35506fbcf5994a0d86ee362df99305e6 P de1e8de15ff04a93a0cfdc98e9312d70: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:21.780182 16937 tablet_bootstrap.cc:492] T 35506fbcf5994a0d86ee362df99305e6 P de1e8de15ff04a93a0cfdc98e9312d70: No bootstrap required, opened a new log
I20260927 10:53:21.780270 16937 ts_tablet_manager.cc:1397] T 35506fbcf5994a0d86ee362df99305e6 P de1e8de15ff04a93a0cfdc98e9312d70: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:53:21.780529 16936 tablet_bootstrap.cc:492] T 35506fbcf5994a0d86ee362df99305e6 P 5cbbac417b1048d4b0139df810615709: No bootstrap required, opened a new log
I20260927 10:53:21.780643 16926 ts_tablet_manager.cc:1918] T 0e4b21981317447b88cf412fe371bc84 P 5cbbac417b1048d4b0139df810615709: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:53:21.780658 16936 ts_tablet_manager.cc:1397] T 35506fbcf5994a0d86ee362df99305e6 P 5cbbac417b1048d4b0139df810615709: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:53:21.780735 16926 log.cc:1196] T 0e4b21981317447b88cf412fe371bc84 P 5cbbac417b1048d4b0139df810615709: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-0/wal/wals/0e4b21981317447b88cf412fe371bc84
I20260927 10:53:21.780736 16937 raft_consensus.cc:348] T 35506fbcf5994a0d86ee362df99305e6 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:21.780867 16937 raft_consensus.cc:374] T 35506fbcf5994a0d86ee362df99305e6 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:21.780913 16937 raft_consensus.cc:729] T 35506fbcf5994a0d86ee362df99305e6 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de1e8de15ff04a93a0cfdc98e9312d70, State: Initialized, Role: FOLLOWER
I20260927 10:53:21.781002 16926 ts_tablet_manager.cc:1939] T 0e4b21981317447b88cf412fe371bc84 P 5cbbac417b1048d4b0139df810615709: Deleting consensus metadata
I20260927 10:53:21.781073 16937 consensus_queue.cc:260] T 35506fbcf5994a0d86ee362df99305e6 P de1e8de15ff04a93a0cfdc98e9312d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:21.781107 16936 raft_consensus.cc:348] T 35506fbcf5994a0d86ee362df99305e6 P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:21.781260 16936 raft_consensus.cc:374] T 35506fbcf5994a0d86ee362df99305e6 P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:21.781304 16936 raft_consensus.cc:729] T 35506fbcf5994a0d86ee362df99305e6 P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cbbac417b1048d4b0139df810615709, State: Initialized, Role: FOLLOWER
I20260927 10:53:21.781353 16937 ts_tablet_manager.cc:1428] T 35506fbcf5994a0d86ee362df99305e6 P de1e8de15ff04a93a0cfdc98e9312d70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:53:21.781433 16936 consensus_queue.cc:260] T 35506fbcf5994a0d86ee362df99305e6 P 5cbbac417b1048d4b0139df810615709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:21.781605 16325 catalog_manager.cc:4841] TS 5cbbac417b1048d4b0139df810615709 (127.14.68.1:45141): tablet 0e4b21981317447b88cf412fe371bc84 (table test-table [id=95b29c86731d4527a42f8643b3504a31]) successfully deleted
I20260927 10:53:21.781733 16936 ts_tablet_manager.cc:1428] T 35506fbcf5994a0d86ee362df99305e6 P 5cbbac417b1048d4b0139df810615709: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:53:21.782243 16939 tablet_bootstrap.cc:492] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c: No bootstrap required, opened a new log
I20260927 10:53:21.782325 16939 ts_tablet_manager.cc:1397] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 10:53:21.782724 16939 raft_consensus.cc:348] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:21.782866 16939 raft_consensus.cc:374] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:21.782894 16939 raft_consensus.cc:729] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f4bc6c7c4d4b9583c043f619299d9c, State: Initialized, Role: FOLLOWER
I20260927 10:53:21.782997 16939 consensus_queue.cc:260] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:21.783326 16939 ts_tablet_manager.cc:1428] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:53:21.982661 16951 raft_consensus.cc:482] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:21.982803 16951 raft_consensus.cc:504] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:21.983276 16951 leader_election.cc:288] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cbbac417b1048d4b0139df810615709 (127.14.68.1:45141), de1e8de15ff04a93a0cfdc98e9312d70 (127.14.68.3:46341)
I20260927 10:53:21.983773 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35506fbcf5994a0d86ee362df99305e6" candidate_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cbbac417b1048d4b0139df810615709" is_pre_election: true
I20260927 10:53:21.983773 16776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35506fbcf5994a0d86ee362df99305e6" candidate_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" is_pre_election: true
I20260927 10:53:21.983925 16776 raft_consensus.cc:2454] T 35506fbcf5994a0d86ee362df99305e6 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6f4bc6c7c4d4b9583c043f619299d9c in term 0.
I20260927 10:53:21.983925 16496 raft_consensus.cc:2454] T 35506fbcf5994a0d86ee362df99305e6 P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6f4bc6c7c4d4b9583c043f619299d9c in term 0.
I20260927 10:53:21.984221 16569 leader_election.cc:302] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cbbac417b1048d4b0139df810615709, e6f4bc6c7c4d4b9583c043f619299d9c; no voters:
I20260927 10:53:21.984385 16951 raft_consensus.cc:2781] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:21.984445 16951 raft_consensus.cc:482] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:21.984483 16951 raft_consensus.cc:3037] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:21.985528 16951 raft_consensus.cc:504] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:21.985859 16951 leader_election.cc:288] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c [CANDIDATE]: Term 1 election: Requested vote from peers 5cbbac417b1048d4b0139df810615709 (127.14.68.1:45141), de1e8de15ff04a93a0cfdc98e9312d70 (127.14.68.3:46341)
I20260927 10:53:21.986232 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35506fbcf5994a0d86ee362df99305e6" candidate_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cbbac417b1048d4b0139df810615709"
I20260927 10:53:21.986223 16776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35506fbcf5994a0d86ee362df99305e6" candidate_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de1e8de15ff04a93a0cfdc98e9312d70"
I20260927 10:53:21.986338 16496 raft_consensus.cc:3037] T 35506fbcf5994a0d86ee362df99305e6 P 5cbbac417b1048d4b0139df810615709 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:21.986366 16776 raft_consensus.cc:3037] T 35506fbcf5994a0d86ee362df99305e6 P de1e8de15ff04a93a0cfdc98e9312d70 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:21.987140 16496 raft_consensus.cc:2454] T 35506fbcf5994a0d86ee362df99305e6 P 5cbbac417b1048d4b0139df810615709 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6f4bc6c7c4d4b9583c043f619299d9c in term 1.
I20260927 10:53:21.987336 16776 raft_consensus.cc:2454] T 35506fbcf5994a0d86ee362df99305e6 P de1e8de15ff04a93a0cfdc98e9312d70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6f4bc6c7c4d4b9583c043f619299d9c in term 1.
I20260927 10:53:21.987408 16569 leader_election.cc:302] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cbbac417b1048d4b0139df810615709, e6f4bc6c7c4d4b9583c043f619299d9c; no voters:
I20260927 10:53:21.987598 16951 raft_consensus.cc:2781] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:21.987751 16951 raft_consensus.cc:686] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c [term 1 LEADER]: Becoming Leader. State: Replica: e6f4bc6c7c4d4b9583c043f619299d9c, State: Running, Role: LEADER
I20260927 10:53:21.987906 16951 consensus_queue.cc:237] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } }
I20260927 10:53:21.989480 16338 catalog_manager.cc:5495] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c reported cstate change: term changed from 0 to 1, leader changed from <none> to e6f4bc6c7c4d4b9583c043f619299d9c (127.14.68.2). New cstate: current_term: 1 leader_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6f4bc6c7c4d4b9583c043f619299d9c" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 } health_report { overall_health: UNKNOWN } } }
I20260927 10:53:22.030365 16776 raft_consensus.cc:1261] T 35506fbcf5994a0d86ee362df99305e6 P de1e8de15ff04a93a0cfdc98e9312d70 [term 1 FOLLOWER]: Refusing update from remote peer e6f4bc6c7c4d4b9583c043f619299d9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:22.030365 16496 raft_consensus.cc:1261] T 35506fbcf5994a0d86ee362df99305e6 P 5cbbac417b1048d4b0139df810615709 [term 1 FOLLOWER]: Refusing update from remote peer e6f4bc6c7c4d4b9583c043f619299d9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:22.030757 16951 consensus_queue.cc:1035] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cbbac417b1048d4b0139df810615709" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:22.030908 16943 consensus_queue.cc:1035] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "de1e8de15ff04a93a0cfdc98e9312d70" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 10:53:22.043882 16910 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 10:53:23.590694 16339 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:23.590845 16339 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:23.593242 16339 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc2a9bdceed2423cb802d0803d5964aa: Sending DeleteTablet for 3 replicas of tablet 35506fbcf5994a0d86ee362df99305e6
I20260927 10:53:23.593997 16476 tablet_service.cc:1514] Processing DeleteTablet for tablet 35506fbcf5994a0d86ee362df99305e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:23 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:56228
I20260927 10:53:23.594024 16616 tablet_service.cc:1514] Processing DeleteTablet for tablet 35506fbcf5994a0d86ee362df99305e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:23 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:41102
I20260927 10:53:23.594388 16983 tablet_replica.cc:330] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c: stopping tablet replica
I20260927 10:53:23.594475 16755 tablet_service.cc:1514] Processing DeleteTablet for tablet 35506fbcf5994a0d86ee362df99305e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:23 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:57128
I20260927 10:53:23.594527 16983 raft_consensus.cc:2229] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:53:23.594703 16983 raft_consensus.cc:2258] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:23.594796 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 16411
I20260927 10:53:23.594835 16984 tablet_replica.cc:330] T 35506fbcf5994a0d86ee362df99305e6 P de1e8de15ff04a93a0cfdc98e9312d70: stopping tablet replica
I20260927 10:53:23.595574 16984 raft_consensus.cc:2229] T 35506fbcf5994a0d86ee362df99305e6 P de1e8de15ff04a93a0cfdc98e9312d70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:23.595743 16984 raft_consensus.cc:2258] T 35506fbcf5994a0d86ee362df99305e6 P de1e8de15ff04a93a0cfdc98e9312d70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:23.596343 16984 ts_tablet_manager.cc:1905] T 35506fbcf5994a0d86ee362df99305e6 P de1e8de15ff04a93a0cfdc98e9312d70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:23.596465 16983 ts_tablet_manager.cc:1905] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:23.598539 16984 ts_tablet_manager.cc:1918] T 35506fbcf5994a0d86ee362df99305e6 P de1e8de15ff04a93a0cfdc98e9312d70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:53:23.598634 16984 log.cc:1196] T 35506fbcf5994a0d86ee362df99305e6 P de1e8de15ff04a93a0cfdc98e9312d70: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/35506fbcf5994a0d86ee362df99305e6
I20260927 10:53:23.598904 16984 ts_tablet_manager.cc:1939] T 35506fbcf5994a0d86ee362df99305e6 P de1e8de15ff04a93a0cfdc98e9312d70: Deleting consensus metadata
I20260927 10:53:23.599534 16324 catalog_manager.cc:4841] TS de1e8de15ff04a93a0cfdc98e9312d70 (127.14.68.3:46341): tablet 35506fbcf5994a0d86ee362df99305e6 (table test-table [id=9af54c98c9be4ea0b0016b7cc1aa76cd]) successfully deleted
I20260927 10:53:23.599658 16983 ts_tablet_manager.cc:1918] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:53:23.599761 16983 log.cc:1196] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/35506fbcf5994a0d86ee362df99305e6
I20260927 10:53:23.600075 16983 ts_tablet_manager.cc:1939] T 35506fbcf5994a0d86ee362df99305e6 P e6f4bc6c7c4d4b9583c043f619299d9c: Deleting consensus metadata
I20260927 10:53:23.600775 16324 catalog_manager.cc:4841] TS e6f4bc6c7c4d4b9583c043f619299d9c (127.14.68.2:37707): tablet 35506fbcf5994a0d86ee362df99305e6 (table test-table [id=9af54c98c9be4ea0b0016b7cc1aa76cd]) successfully deleted
W20260927 10:53:23.602813 16325 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.68.1:45141 (error 108)
I20260927 10:53:23.603531 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 16551
I20260927 10:53:23.605345 16986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:23.603534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:45141 (local address 127.0.0.1:39154)
0927 10:53:23.605202 (+ 1668us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:23.605267 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.1:45141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1587,"thread_start_us":93,"threads_started":1}
W20260927 10:53:23.605464 16325 catalog_manager.cc:4584] TS 5cbbac417b1048d4b0139df810615709 (127.14.68.1:45141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 35506fbcf5994a0d86ee362df99305e6: Network error: Client connection negotiation failed: client connection to 127.14.68.1:45141: connect: Connection refused (error 111)
I20260927 10:53:23.610325 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 16691
I20260927 10:53:23.616768 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 16307
2026-09-27T10:53:23Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6737 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:53:23 dist-test-slave-tz51 krb5kdc[16989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:53:23 dist-test-slave-tz51 krb5kdc[16989](info): set up 2 sockets
Sep 27 10:53:23 dist-test-slave-tz51 krb5kdc[16989](info): commencing operation
krb5kdc: starting...
W20260927 10:53:24.651765 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:53:24 dist-test-slave-tz51 krb5kdc[16989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:53:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:53:24.771070 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:39755
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:37351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:39755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:24.883486 17002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:24.883822 17002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:24.883939 17002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:24.887988 17002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:24.888077 17002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:24.888118 17002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:24.888176 17002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:24.888212 17002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:24.893347 17002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:37351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:39755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:39755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:24.894500 17002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:24.895514 17002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:53:24.904243 17002 server_base.cc:1034] running on GCE node
W20260927 10:53:24.904258 17007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:24.904275 17008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:24.904418 17010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:24.904951 17002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:24.906033 17002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:24.907260 17002 hybrid_clock.cc:648] HybridClock initialized: now 1790506404907247 us; error 24 us; skew 500 ppm
Sep 27 10:53:24 dist-test-slave-tz51 krb5kdc[16989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506404, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:24.910005 17002 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:53:24.910866 17002 webserver.cc:460] Webserver started at http://127.14.68.62:43161/ using document root <none> and password file <none>
I20260927 10:53:24.911422 17002 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:24.911512 17002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:24.911793 17002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:24.913887 17002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "32b381eebd51476396cd829f02a9d3ba"
format_stamp: "Formatted at 2026-09-27 10:53:24 on dist-test-slave-tz51"
server_key: "3fae85cd090c398e2a7c4961f8e56e96"
server_key_iv: "e09ad8ef31c9a0adce464edd4ef7580f"
server_key_version: "encryptionkey@0"
I20260927 10:53:24.914436 17002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "32b381eebd51476396cd829f02a9d3ba"
format_stamp: "Formatted at 2026-09-27 10:53:24 on dist-test-slave-tz51"
server_key: "3fae85cd090c398e2a7c4961f8e56e96"
server_key_iv: "e09ad8ef31c9a0adce464edd4ef7580f"
server_key_version: "encryptionkey@0"
I20260927 10:53:24.917945 17002 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:53:24.920255 17017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:24.921416 17002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:24.921545 17002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "32b381eebd51476396cd829f02a9d3ba"
format_stamp: "Formatted at 2026-09-27 10:53:24 on dist-test-slave-tz51"
server_key: "3fae85cd090c398e2a7c4961f8e56e96"
server_key_iv: "e09ad8ef31c9a0adce464edd4ef7580f"
server_key_version: "encryptionkey@0"
I20260927 10:53:24.921653 17002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:24.933305 17002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:24.936427 17002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:24.936623 17002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:24.943600 17002 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:39755
I20260927 10:53:24.943677 17069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:39755 every 8 connection(s)
I20260927 10:53:24.944633 17002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:24.947000 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 17002
I20260927 10:53:24.947115 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:53:24.947404 14608 external_mini_cluster.cc:1428] Setting key 1584afe7232613a40056634bd2cf44bc
I20260927 10:53:24.947662 17070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:24.954744 17070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32b381eebd51476396cd829f02a9d3ba: Bootstrap starting.
Sep 27 10:53:24 dist-test-slave-tz51 krb5kdc[16989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:24.957427 17070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32b381eebd51476396cd829f02a9d3ba: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:24.958109 17070 log.cc:824] T 00000000000000000000000000000000 P 32b381eebd51476396cd829f02a9d3ba: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:24.960052 17070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32b381eebd51476396cd829f02a9d3ba: No bootstrap required, opened a new log
I20260927 10:53:24.961786 17073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:24.948766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43628 (local address 127.14.68.62:39755)
0927 10:53:24.949160 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:24.949184 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:24.949220 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:53:24.949860 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:24.949877 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:24.950195 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:24.950489 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:24.951435 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:24.952487 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:24.953702 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:24.954148 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:24.954260 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:24.957028 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:24.957035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:24.957050 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:24.957081 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:24.958878 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:24.959339 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:24.959344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:24.959348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:24.959455 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:24.959790 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:24.959794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:24.959799 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:24.960255 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:24.960451 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:24.960737 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 10:53:24.961056 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":141,"threads_started":1}
I20260927 10:53:24.964046 17070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32b381eebd51476396cd829f02a9d3ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32b381eebd51476396cd829f02a9d3ba" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 39755 } }
I20260927 10:53:24.964349 17070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32b381eebd51476396cd829f02a9d3ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:24.964432 17070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32b381eebd51476396cd829f02a9d3ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32b381eebd51476396cd829f02a9d3ba, State: Initialized, Role: FOLLOWER
I20260927 10:53:24.964883 17070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32b381eebd51476396cd829f02a9d3ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32b381eebd51476396cd829f02a9d3ba" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 39755 } }
I20260927 10:53:24.965032 17070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32b381eebd51476396cd829f02a9d3ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:24.965101 17070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32b381eebd51476396cd829f02a9d3ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:24.965200 17070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32b381eebd51476396cd829f02a9d3ba [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:24.966310 17070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32b381eebd51476396cd829f02a9d3ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32b381eebd51476396cd829f02a9d3ba" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 39755 } }
I20260927 10:53:24.966676 17070 leader_election.cc:302] T 00000000000000000000000000000000 P 32b381eebd51476396cd829f02a9d3ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32b381eebd51476396cd829f02a9d3ba; no voters:
I20260927 10:53:24.966992 17070 leader_election.cc:288] T 00000000000000000000000000000000 P 32b381eebd51476396cd829f02a9d3ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:24.967147 17075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32b381eebd51476396cd829f02a9d3ba [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:24.967407 17075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32b381eebd51476396cd829f02a9d3ba [term 1 LEADER]: Becoming Leader. State: Replica: 32b381eebd51476396cd829f02a9d3ba, State: Running, Role: LEADER
I20260927 10:53:24.967764 17075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32b381eebd51476396cd829f02a9d3ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32b381eebd51476396cd829f02a9d3ba" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 39755 } }
I20260927 10:53:24.968317 17070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32b381eebd51476396cd829f02a9d3ba [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:24.969480 17077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32b381eebd51476396cd829f02a9d3ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32b381eebd51476396cd829f02a9d3ba. Latest consensus state: current_term: 1 leader_uuid: "32b381eebd51476396cd829f02a9d3ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32b381eebd51476396cd829f02a9d3ba" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 39755 } } }
I20260927 10:53:24.969601 17077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32b381eebd51476396cd829f02a9d3ba [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:24.969981 17084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:24.969969 17076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32b381eebd51476396cd829f02a9d3ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32b381eebd51476396cd829f02a9d3ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32b381eebd51476396cd829f02a9d3ba" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 39755 } } }
I20260927 10:53:24.970085 17076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32b381eebd51476396cd829f02a9d3ba [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:24.973301 17084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:24.978927 17084 catalog_manager.cc:1347] Generated new cluster ID: f8c89af46cb24f8f8d12f44ece1b6bda
I20260927 10:53:24.979043 17084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:24.998073 17084 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:24.999356 17084 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:25.005697 17084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32b381eebd51476396cd829f02a9d3ba: Generated new TSK 0
I20260927 10:53:25.006564 17084 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
I20260927 10:53:25.043303 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:39755
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:37351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:25.152046 17096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:25.152284 17096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:25.152343 17096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:25.156194 17096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:25.156235 17096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:25.156287 17096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:53:25.161311 17096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:37351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:39755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:25.162233 17096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:25.163144 17096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:25.172525 17102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:25.172533 17101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:25.172533 17104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:25.172832 17096 server_base.cc:1034] running on GCE node
I20260927 10:53:25.173167 17096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:25.173746 17096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:25.174898 17096 hybrid_clock.cc:648] HybridClock initialized: now 1790506405174868 us; error 46 us; skew 500 ppm
Sep 27 10:53:25 dist-test-slave-tz51 krb5kdc[16989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506405, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:25.177587 17096 init.cc:377] Logged in from keytab as kudu/127.14.68.1@KRBTEST.COM (short username kudu)
I20260927 10:53:25.178453 17096 webserver.cc:460] Webserver started at http://127.14.68.1:34015/ using document root <none> and password file <none>
I20260927 10:53:25.179013 17096 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:25.179101 17096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:25.179345 17096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:25.181437 17096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "cb1d8d805f844cf393c9d368db20b430"
format_stamp: "Formatted at 2026-09-27 10:53:25 on dist-test-slave-tz51"
server_key: "67daae75a48183a9350e288006b20850"
server_key_iv: "21c35eed0bb30296cfc997a2a6e152d0"
server_key_version: "encryptionkey@0"
I20260927 10:53:25.181999 17096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "cb1d8d805f844cf393c9d368db20b430"
format_stamp: "Formatted at 2026-09-27 10:53:25 on dist-test-slave-tz51"
server_key: "67daae75a48183a9350e288006b20850"
server_key_iv: "21c35eed0bb30296cfc997a2a6e152d0"
server_key_version: "encryptionkey@0"
I20260927 10:53:25.185693 17096 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:53:25.188176 17111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:25.189457 17096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:25.189585 17096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "cb1d8d805f844cf393c9d368db20b430"
format_stamp: "Formatted at 2026-09-27 10:53:25 on dist-test-slave-tz51"
server_key: "67daae75a48183a9350e288006b20850"
server_key_iv: "21c35eed0bb30296cfc997a2a6e152d0"
server_key_version: "encryptionkey@0"
I20260927 10:53:25.189692 17096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:25.203106 17096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:25.206265 17096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:25.206480 17096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:25.207078 17096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:25.207971 17096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:25.208040 17096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:25.208104 17096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:25.208146 17096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:25.217171 17096 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:41237
I20260927 10:53:25.217195 17224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:41237 every 8 connection(s)
I20260927 10:53:25.218063 17096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:53:25.219224 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 17096
I20260927 10:53:25.219313 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:53:25.219599 14608 external_mini_cluster.cc:1428] Setting key 4df0845f8eaba9831f2402aa2c98227a
Sep 27 10:53:25 dist-test-slave-tz51 krb5kdc[16989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506405, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.2@KRBTEST.COM" created.
I20260927 10:53:25.233598 17073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:25.220454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:43157 (local address 127.14.68.62:39755)
0927 10:53:25.220608 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:25.220613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:25.221701 (+ 1088us) server_negotiation.cc:407] Connection header received
0927 10:53:25.223065 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:25.223071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:25.223139 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:25.223234 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:25.224484 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:25.225470 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:25.226676 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:25.227135 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:25.227244 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:25.229875 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:25.229878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:25.229881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:25.229903 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:25.231293 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:25.231850 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:25.231854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:25.231856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:25.231967 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:25.232420 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:25.232423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:25.232424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:25.232600 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:25.232679 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:25.233363 (+ 684us) server_negotiation.cc:299] Negotiation successful
0927 10:53:25.233471 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:53:25.234870 17227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:25.220729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:39755 (local address 127.14.68.1:43157)
0927 10:53:25.221510 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:25.221593 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:25.222810 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:25.223395 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:25.223402 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:25.223864 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:25.224351 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:25.224366 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:25.225597 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:25.225600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:25.226548 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:25.226558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:25.227289 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:25.227292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:25.227432 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:25.228051 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:25.228067 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:25.229705 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:25.231416 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:25.231420 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:25.231437 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:25.231689 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:25.232132 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:25.232134 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:25.232136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:25.232305 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:25.232721 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:25.232731 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:25.233085 (+ 354us) client_negotiation.cc:769] Sending connection context
0927 10:53:25.233294 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 10:53:25.233672 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":145,"threads_started":1}
I20260927 10:53:25.236245 17225 heartbeater.cc:347] Connected to a master server at 127.14.68.62:39755
I20260927 10:53:25.236508 17225 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:25.237020 17225 heartbeater.cc:511] Master 127.14.68.62:39755 requested a full tablet report, sending...
I20260927 10:53:25.238529 17034 ts_manager.cc:194] Registered new tserver with Master: cb1d8d805f844cf393c9d368db20b430 (127.14.68.1:41237)
I20260927 10:53:25.239794 17034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:43157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
I20260927 10:53:25.248484 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:39755
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:37351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:25.357753 17230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:25.358013 17230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:25.358057 17230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:25.361989 17230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:25.362049 17230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:25.362108 17230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:53:25.367430 17230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:37351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:39755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:25.368664 17230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:25.369757 17230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:25.379277 17236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:25.379338 17235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:25.379487 17230 server_base.cc:1034] running on GCE node
W20260927 10:53:25.379563 17238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:25.379972 17230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:25.380602 17230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:25.381783 17230 hybrid_clock.cc:648] HybridClock initialized: now 1790506405381751 us; error 45 us; skew 500 ppm
Sep 27 10:53:25 dist-test-slave-tz51 krb5kdc[16989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506405, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:25.384392 17230 init.cc:377] Logged in from keytab as kudu/127.14.68.2@KRBTEST.COM (short username kudu)
I20260927 10:53:25.385232 17230 webserver.cc:460] Webserver started at http://127.14.68.2:40311/ using document root <none> and password file <none>
I20260927 10:53:25.385797 17230 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:25.385890 17230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:25.386137 17230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:25.388199 17230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "f2726a78b2c94a24969b814961f0fa99"
format_stamp: "Formatted at 2026-09-27 10:53:25 on dist-test-slave-tz51"
server_key: "d41cd8b3e06bab8da4eb9a8c0d99ee68"
server_key_iv: "eba00d944f2b4819b10405149fd88c4d"
server_key_version: "encryptionkey@0"
I20260927 10:53:25.388751 17230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "f2726a78b2c94a24969b814961f0fa99"
format_stamp: "Formatted at 2026-09-27 10:53:25 on dist-test-slave-tz51"
server_key: "d41cd8b3e06bab8da4eb9a8c0d99ee68"
server_key_iv: "eba00d944f2b4819b10405149fd88c4d"
server_key_version: "encryptionkey@0"
I20260927 10:53:25.392391 17230 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:53:25.394786 17245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:25.396095 17230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:25.396234 17230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "f2726a78b2c94a24969b814961f0fa99"
format_stamp: "Formatted at 2026-09-27 10:53:25 on dist-test-slave-tz51"
server_key: "d41cd8b3e06bab8da4eb9a8c0d99ee68"
server_key_iv: "eba00d944f2b4819b10405149fd88c4d"
server_key_version: "encryptionkey@0"
I20260927 10:53:25.396340 17230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:25.417472 17230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:25.420395 17230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:25.420603 17230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:25.421190 17230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:25.421994 17230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:25.422067 17230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:25.422147 17230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:25.422183 17230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:25.430912 17230 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:42355
I20260927 10:53:25.430938 17358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:42355 every 8 connection(s)
I20260927 10:53:25.431875 17230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:53:25.434363 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 17230
I20260927 10:53:25.434455 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:53:25.434752 14608 external_mini_cluster.cc:1428] Setting key fe36f299ca4181a78ec1b0a627b3c442
Sep 27 10:53:25 dist-test-slave-tz51 krb5kdc[16989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506405, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.3@KRBTEST.COM" created.
I20260927 10:53:25.447180 17073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:25.433872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:54259 (local address 127.14.68.62:39755)
0927 10:53:25.434018 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:25.434022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:25.435256 (+ 1234us) server_negotiation.cc:407] Connection header received
0927 10:53:25.436427 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:25.436431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:25.436487 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:25.436606 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:25.437909 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:25.438502 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:25.440071 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:25.440319 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:25.440445 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:25.443201 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:25.443205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:25.443207 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:25.443233 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:25.444882 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:25.445397 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:25.445402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:25.445404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:25.445486 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:25.445904 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:25.445908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:25.445913 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:25.446160 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:25.446275 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:25.446870 (+ 595us) server_negotiation.cc:299] Negotiation successful
0927 10:53:25.447002 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:53:25.448096 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:25.434241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:39755 (local address 127.14.68.2:54259)
0927 10:53:25.435065 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:25.435123 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:25.436185 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:25.436799 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:25.436807 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:25.437199 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:25.437744 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:25.437759 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:25.438647 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:25.438651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:25.439919 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:25.439930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:25.440489 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:25.440494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:25.440697 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:25.441392 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:25.441411 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:25.443003 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:25.445001 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:25.445007 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:25.445023 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:25.445287 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:25.445586 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:25.445589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:25.445591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:25.445743 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:25.446273 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:25.446279 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:25.446584 (+ 305us) client_negotiation.cc:769] Sending connection context
0927 10:53:25.446831 (+ 247us) client_negotiation.cc:240] Negotiation successful
0927 10:53:25.447176 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":160,"threads_started":1}
I20260927 10:53:25.449390 17359 heartbeater.cc:347] Connected to a master server at 127.14.68.62:39755
I20260927 10:53:25.449659 17359 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:25.450161 17359 heartbeater.cc:511] Master 127.14.68.62:39755 requested a full tablet report, sending...
I20260927 10:53:25.451229 17034 ts_manager.cc:194] Registered new tserver with Master: f2726a78b2c94a24969b814961f0fa99 (127.14.68.2:42355)
I20260927 10:53:25.451879 17034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:54259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
I20260927 10:53:25.462538 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:39755
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:37351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:25.572111 17364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:25.572335 17364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:25.572388 17364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:25.576416 17364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:25.576471 17364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:25.576527 17364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:53:25.581604 17364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:37351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:39755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:25.582520 17364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:25.583438 17364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:25.592993 17372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:25.592985 17369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:25.593056 17364 server_base.cc:1034] running on GCE node
W20260927 10:53:25.592993 17370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:25.593597 17364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:25.594255 17364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:25.595407 17364 hybrid_clock.cc:648] HybridClock initialized: now 1790506405595377 us; error 43 us; skew 500 ppm
Sep 27 10:53:25 dist-test-slave-tz51 krb5kdc[16989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506405, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:25.598098 17364 init.cc:377] Logged in from keytab as kudu/127.14.68.3@KRBTEST.COM (short username kudu)
I20260927 10:53:25.598875 17364 webserver.cc:460] Webserver started at http://127.14.68.3:41235/ using document root <none> and password file <none>
I20260927 10:53:25.599466 17364 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:25.599556 17364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:25.599843 17364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:25.601910 17364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "6edeca1a705d4fc28b1108bb94632cdb"
format_stamp: "Formatted at 2026-09-27 10:53:25 on dist-test-slave-tz51"
server_key: "d75f3d647a5442f102520a12a42d9bc7"
server_key_iv: "5ba9fcac81ffdcb61a4e8308138667a6"
server_key_version: "encryptionkey@0"
I20260927 10:53:25.602466 17364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "6edeca1a705d4fc28b1108bb94632cdb"
format_stamp: "Formatted at 2026-09-27 10:53:25 on dist-test-slave-tz51"
server_key: "d75f3d647a5442f102520a12a42d9bc7"
server_key_iv: "5ba9fcac81ffdcb61a4e8308138667a6"
server_key_version: "encryptionkey@0"
I20260927 10:53:25.606104 17364 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:53:25.608453 17379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:25.609645 17364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:25.609774 17364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "6edeca1a705d4fc28b1108bb94632cdb"
format_stamp: "Formatted at 2026-09-27 10:53:25 on dist-test-slave-tz51"
server_key: "d75f3d647a5442f102520a12a42d9bc7"
server_key_iv: "5ba9fcac81ffdcb61a4e8308138667a6"
server_key_version: "encryptionkey@0"
I20260927 10:53:25.609880 17364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:25.619570 17364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:25.622164 17364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:25.622350 17364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:25.622893 17364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:25.623661 17364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:25.623750 17364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:25.623834 17364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:25.623885 17364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:25.632609 17364 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:45285
I20260927 10:53:25.632645 17492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:45285 every 8 connection(s)
I20260927 10:53:25.633440 17364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:53:25.638564 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 17364
I20260927 10:53:25.638674 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:53:25.638942 14608 external_mini_cluster.cc:1428] Setting key fd75174e507e68db287820388e07b1ed
Sep 27 10:53:25 dist-test-slave-tz51 krb5kdc[16989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506405, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:25.646435 17073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:25.635173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:57535 (local address 127.14.68.62:39755)
0927 10:53:25.635328 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:25.635332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:25.636279 (+ 947us) server_negotiation.cc:407] Connection header received
0927 10:53:25.637148 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:25.637152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:25.637212 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:25.637318 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:25.638510 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:25.639245 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:25.640688 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:25.641016 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:25.641113 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:25.643468 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:25.643471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:25.643473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:25.643495 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:25.644735 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:25.645185 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:25.645187 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:25.645189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:25.645258 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:25.645532 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:25.645534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:25.645535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:25.645693 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:25.645764 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:25.646222 (+ 458us) server_negotiation.cc:299] Negotiation successful
0927 10:53:25.646315 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 10:53:25.647249 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:25.635478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:39755 (local address 127.14.68.3:57535)
0927 10:53:25.636129 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:25.636168 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:25.636947 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:25.637483 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:25.637490 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:25.637831 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:25.638330 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:25.638352 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:25.639387 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:25.639411 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:25.640572 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:25.640581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:25.641157 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:25.641160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:25.641308 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:25.641905 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:25.641921 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:25.643327 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:25.644834 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:25.644839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:25.644850 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:25.645091 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:25.645334 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:25.645336 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:25.645337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:25.645458 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:25.645775 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:25.645779 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:25.645992 (+ 213us) client_negotiation.cc:769] Sending connection context
0927 10:53:25.646161 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 10:53:25.646412 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":107,"threads_started":1}
I20260927 10:53:25.648416 17493 heartbeater.cc:347] Connected to a master server at 127.14.68.62:39755
I20260927 10:53:25.648670 17493 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:25.649184 17493 heartbeater.cc:511] Master 127.14.68.62:39755 requested a full tablet report, sending...
I20260927 10:53:25.650197 17034 ts_manager.cc:194] Registered new tserver with Master: 6edeca1a705d4fc28b1108bb94632cdb (127.14.68.3:45285)
I20260927 10:53:25.650784 17034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:57535
I20260927 10:53:25.652822 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 10:53:25.665226 17073 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:25.663906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43640 (local address 127.14.68.62:39755)
0927 10:53:25.664036 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:25.664043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:25.664207 (+ 164us) server_negotiation.cc:407] Connection header received
0927 10:53:25.664318 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:25.664320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:25.664363 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:25.664436 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:25.665098 (+ 662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43640: BlockingRecv error: recv got EOF from 127.0.0.1:43640 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:53:25.666328 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 17096
I20260927 10:53:25.671860 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 17230
I20260927 10:53:25.677326 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 17364
I20260927 10:53:25.682440 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 17002
2026-09-27T10:53:25Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2064 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:53:25 dist-test-slave-tz51 krb5kdc[17505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:53:25 dist-test-slave-tz51 krb5kdc[17505](info): set up 2 sockets
Sep 27 10:53:25 dist-test-slave-tz51 krb5kdc[17505](info): commencing operation
krb5kdc: starting...
W20260927 10:53:26.719645 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:53:26 dist-test-slave-tz51 krb5kdc[17505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:53:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:53:26.840401 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:36717
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:33359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:36717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:26.947839 17518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:26.948101 17518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:26.948141 17518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:26.952260 17518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:26.952318 17518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:26.952338 17518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:26.952392 17518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:26.952422 17518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:26.957592 17518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:33359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:36717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:36717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:26.958612 17518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:26.959635 17518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:26.968159 17523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:26.968147 17524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:26.968307 17518 server_base.cc:1034] running on GCE node
W20260927 10:53:26.968225 17526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:26.968990 17518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:26.970134 17518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:26.971323 17518 hybrid_clock.cc:648] HybridClock initialized: now 1790506406971289 us; error 50 us; skew 500 ppm
Sep 27 10:53:26 dist-test-slave-tz51 krb5kdc[17505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506406, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:26.979104 17518 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:53:26.980172 17518 webserver.cc:460] Webserver started at http://127.14.68.62:43871/ using document root <none> and password file <none>
I20260927 10:53:26.980767 17518 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:26.980858 17518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:26.981099 17518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:26.983188 17518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "95078d24c3a24a09b778320b7b3c3bbf"
format_stamp: "Formatted at 2026-09-27 10:53:26 on dist-test-slave-tz51"
server_key: "1efb261df24f15b7af658e370367cab7"
server_key_iv: "ceeb188cc2acec6775dd08cc151689ed"
server_key_version: "encryptionkey@0"
I20260927 10:53:26.983806 17518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "95078d24c3a24a09b778320b7b3c3bbf"
format_stamp: "Formatted at 2026-09-27 10:53:26 on dist-test-slave-tz51"
server_key: "1efb261df24f15b7af658e370367cab7"
server_key_iv: "ceeb188cc2acec6775dd08cc151689ed"
server_key_version: "encryptionkey@0"
I20260927 10:53:26.987488 17518 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:53:26.989908 17533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:26.991357 17518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:26.991526 17518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "95078d24c3a24a09b778320b7b3c3bbf"
format_stamp: "Formatted at 2026-09-27 10:53:26 on dist-test-slave-tz51"
server_key: "1efb261df24f15b7af658e370367cab7"
server_key_iv: "ceeb188cc2acec6775dd08cc151689ed"
server_key_version: "encryptionkey@0"
I20260927 10:53:26.991650 17518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:27.006664 17518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:27.009502 17518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:27.009709 17518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:27.016595 17518 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:36717
I20260927 10:53:27.016875 17585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:36717 every 8 connection(s)
I20260927 10:53:27.017624 17518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:27.020391 17586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:27.025744 17586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95078d24c3a24a09b778320b7b3c3bbf: Bootstrap starting.
I20260927 10:53:27.026376 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 17518
I20260927 10:53:27.026508 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:53:27.026772 14608 external_mini_cluster.cc:1428] Setting key 34d10c37d8653f9d854fa41d294de09d
I20260927 10:53:27.028525 17586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95078d24c3a24a09b778320b7b3c3bbf: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:27.029397 17586 log.cc:824] T 00000000000000000000000000000000 P 95078d24c3a24a09b778320b7b3c3bbf: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:27.031426 17586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95078d24c3a24a09b778320b7b3c3bbf: No bootstrap required, opened a new log
I20260927 10:53:27.035526 17586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95078d24c3a24a09b778320b7b3c3bbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95078d24c3a24a09b778320b7b3c3bbf" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 36717 } }
I20260927 10:53:27.035840 17586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95078d24c3a24a09b778320b7b3c3bbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:27.035908 17586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95078d24c3a24a09b778320b7b3c3bbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95078d24c3a24a09b778320b7b3c3bbf, State: Initialized, Role: FOLLOWER
Sep 27 10:53:27 dist-test-slave-tz51 krb5kdc[17505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:27.036419 17586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95078d24c3a24a09b778320b7b3c3bbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95078d24c3a24a09b778320b7b3c3bbf" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 36717 } }
I20260927 10:53:27.036593 17586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95078d24c3a24a09b778320b7b3c3bbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:27.036662 17586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95078d24c3a24a09b778320b7b3c3bbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:27.036794 17586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95078d24c3a24a09b778320b7b3c3bbf [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:27.037933 17586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95078d24c3a24a09b778320b7b3c3bbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95078d24c3a24a09b778320b7b3c3bbf" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 36717 } }
I20260927 10:53:27.038288 17586 leader_election.cc:302] T 00000000000000000000000000000000 P 95078d24c3a24a09b778320b7b3c3bbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95078d24c3a24a09b778320b7b3c3bbf; no voters:
I20260927 10:53:27.038584 17586 leader_election.cc:288] T 00000000000000000000000000000000 P 95078d24c3a24a09b778320b7b3c3bbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:27.038705 17591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95078d24c3a24a09b778320b7b3c3bbf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:27.038916 17591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95078d24c3a24a09b778320b7b3c3bbf [term 1 LEADER]: Becoming Leader. State: Replica: 95078d24c3a24a09b778320b7b3c3bbf, State: Running, Role: LEADER
I20260927 10:53:27.039253 17591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95078d24c3a24a09b778320b7b3c3bbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95078d24c3a24a09b778320b7b3c3bbf" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 36717 } }
I20260927 10:53:27.039986 17586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95078d24c3a24a09b778320b7b3c3bbf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:27.040534 17593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95078d24c3a24a09b778320b7b3c3bbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95078d24c3a24a09b778320b7b3c3bbf. Latest consensus state: current_term: 1 leader_uuid: "95078d24c3a24a09b778320b7b3c3bbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95078d24c3a24a09b778320b7b3c3bbf" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 36717 } } }
I20260927 10:53:27.040655 17593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95078d24c3a24a09b778320b7b3c3bbf [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:27.040984 17592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95078d24c3a24a09b778320b7b3c3bbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95078d24c3a24a09b778320b7b3c3bbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95078d24c3a24a09b778320b7b3c3bbf" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 36717 } } }
I20260927 10:53:27.041079 17592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95078d24c3a24a09b778320b7b3c3bbf [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:27.041347 17597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:27.042970 17589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:27.028098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51442 (local address 127.14.68.62:36717)
0927 10:53:27.028527 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:27.028547 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:27.028590 (+ 43us) server_negotiation.cc:407] Connection header received
0927 10:53:27.029440 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:27.029464 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:27.029868 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:27.030207 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:27.030864 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:27.032320 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:27.033150 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:27.033854 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:27.034036 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:27.036840 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:27.036846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:27.036858 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:27.036884 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:27.038867 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:27.039410 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:27.039416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:27.039422 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:27.039553 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:27.039902 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:27.039905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:27.039907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:27.040480 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:27.040668 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:27.041564 (+ 896us) server_negotiation.cc:299] Negotiation successful
0927 10:53:27.041912 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":114,"threads_started":1}
I20260927 10:53:27.045603 17597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:27.046010 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:53:27.051499 17597 catalog_manager.cc:1347] Generated new cluster ID: b897c583dced4f2381fd8879c0be953e
I20260927 10:53:27.051563 17597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:27.065414 17597 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:27.066634 17597 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:27.072278 17589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:27.066517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51448 (local address 127.14.68.62:36717)
0927 10:53:27.066633 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:27.066637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:27.066942 (+ 305us) server_negotiation.cc:407] Connection header received
0927 10:53:27.067072 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:27.067076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:27.067128 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:27.067225 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:27.067830 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:27.068617 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:27.069852 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:27.070195 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:27.070282 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:27.071227 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:27.071230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:27.071233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:27.071818 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:27.071849 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:27.071869 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:27.071954 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:27.072079 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 10:53:27.072137 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:53:27.081619 17597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95078d24c3a24a09b778320b7b3c3bbf: Generated new TSK 0
I20260927 10:53:27.082439 17597 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:53:27.090871 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 17518
2026-09-27T10:53:27Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1406 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:53:27 dist-test-slave-tz51 krb5kdc[17619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:53:27 dist-test-slave-tz51 krb5kdc[17619](info): set up 2 sockets
Sep 27 10:53:27 dist-test-slave-tz51 krb5kdc[17619](info): commencing operation
krb5kdc: starting...
W20260927 10:53:28.111830 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:53:28 dist-test-slave-tz51 krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:53:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:53:28.228543 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:33035
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:33035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:28.334838 17632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:28.335103 17632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:28.335180 17632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:28.339169 17632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:28.339251 17632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:28.339287 17632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:28.339342 17632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:28.339394 17632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:28.344611 17632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:33035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:33035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:28.345615 17632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:28.346611 17632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:28.354884 17638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:28.354956 17632 server_base.cc:1034] running on GCE node
W20260927 10:53:28.354894 17640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:28.354894 17637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:28.355880 17632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:28.357496 17632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:28.358700 17632 hybrid_clock.cc:648] HybridClock initialized: now 1790506408358664 us; error 47 us; skew 500 ppm
Sep 27 10:53:28 dist-test-slave-tz51 krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506408, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:28.361507 17632 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:53:28.362349 17632 webserver.cc:460] Webserver started at http://127.14.68.62:35515/ using document root <none> and password file <none>
I20260927 10:53:28.362906 17632 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:28.362994 17632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:28.363216 17632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:28.365284 17632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "bc84aa3ccd0643209f8cd71e5851d315"
format_stamp: "Formatted at 2026-09-27 10:53:28 on dist-test-slave-tz51"
server_key: "83682ad1d924ecd8ab7bb25c1b2936f6"
server_key_iv: "de00e0259261050c9671313fe38738a1"
server_key_version: "encryptionkey@0"
I20260927 10:53:28.365830 17632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "bc84aa3ccd0643209f8cd71e5851d315"
format_stamp: "Formatted at 2026-09-27 10:53:28 on dist-test-slave-tz51"
server_key: "83682ad1d924ecd8ab7bb25c1b2936f6"
server_key_iv: "de00e0259261050c9671313fe38738a1"
server_key_version: "encryptionkey@0"
I20260927 10:53:28.369376 17632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:53:28.371551 17647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:28.372779 17632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:28.372916 17632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "bc84aa3ccd0643209f8cd71e5851d315"
format_stamp: "Formatted at 2026-09-27 10:53:28 on dist-test-slave-tz51"
server_key: "83682ad1d924ecd8ab7bb25c1b2936f6"
server_key_iv: "de00e0259261050c9671313fe38738a1"
server_key_version: "encryptionkey@0"
I20260927 10:53:28.373023 17632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:28.387406 17632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:28.390115 17632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:28.390352 17632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:28.396764 17632 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:33035
I20260927 10:53:28.397042 17699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:33035 every 8 connection(s)
I20260927 10:53:28.397684 17632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:28.400538 17700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:28.403761 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 17632
I20260927 10:53:28.403894 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:53:28.404155 14608 external_mini_cluster.cc:1428] Setting key a94200fbf30ec6f28151987631031cdc
I20260927 10:53:28.406184 17700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc84aa3ccd0643209f8cd71e5851d315: Bootstrap starting.
I20260927 10:53:28.408671 17700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc84aa3ccd0643209f8cd71e5851d315: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:28.409518 17700 log.cc:824] T 00000000000000000000000000000000 P bc84aa3ccd0643209f8cd71e5851d315: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:28.411422 17700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc84aa3ccd0643209f8cd71e5851d315: No bootstrap required, opened a new log
Sep 27 10:53:28 dist-test-slave-tz51 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:28.414593 17700 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc84aa3ccd0643209f8cd71e5851d315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc84aa3ccd0643209f8cd71e5851d315" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33035 } }
I20260927 10:53:28.414820 17700 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc84aa3ccd0643209f8cd71e5851d315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:28.414881 17700 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc84aa3ccd0643209f8cd71e5851d315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc84aa3ccd0643209f8cd71e5851d315, State: Initialized, Role: FOLLOWER
I20260927 10:53:28.415381 17700 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc84aa3ccd0643209f8cd71e5851d315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc84aa3ccd0643209f8cd71e5851d315" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33035 } }
I20260927 10:53:28.415532 17700 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc84aa3ccd0643209f8cd71e5851d315 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:28.415599 17700 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc84aa3ccd0643209f8cd71e5851d315 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:28.415724 17700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc84aa3ccd0643209f8cd71e5851d315 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:28.416733 17700 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc84aa3ccd0643209f8cd71e5851d315 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc84aa3ccd0643209f8cd71e5851d315" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33035 } }
I20260927 10:53:28.417078 17700 leader_election.cc:302] T 00000000000000000000000000000000 P bc84aa3ccd0643209f8cd71e5851d315 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc84aa3ccd0643209f8cd71e5851d315; no voters:
I20260927 10:53:28.417410 17700 leader_election.cc:288] T 00000000000000000000000000000000 P bc84aa3ccd0643209f8cd71e5851d315 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:28.417502 17705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc84aa3ccd0643209f8cd71e5851d315 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:28.417757 17705 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc84aa3ccd0643209f8cd71e5851d315 [term 1 LEADER]: Becoming Leader. State: Replica: bc84aa3ccd0643209f8cd71e5851d315, State: Running, Role: LEADER
I20260927 10:53:28.418118 17705 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc84aa3ccd0643209f8cd71e5851d315 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc84aa3ccd0643209f8cd71e5851d315" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33035 } }
I20260927 10:53:28.418672 17700 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc84aa3ccd0643209f8cd71e5851d315 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:28.418906 17703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:28.405435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60148 (local address 127.14.68.62:33035)
0927 10:53:28.405823 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:28.405837 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:28.405872 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:53:28.406532 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:28.406549 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:28.406864 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:28.407134 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:28.408041 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:28.409092 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:28.410194 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:28.410651 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:28.410828 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:28.413455 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:28.413460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:28.413472 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:28.413497 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:28.415929 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:28.416337 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:28.416343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:28.416349 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:28.416488 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:28.416746 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:28.416751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:28.416753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:28.417229 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:28.417456 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:28.417734 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 10:53:28.418011 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":129,"threads_started":1}
I20260927 10:53:28.419791 17707 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc84aa3ccd0643209f8cd71e5851d315 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc84aa3ccd0643209f8cd71e5851d315. Latest consensus state: current_term: 1 leader_uuid: "bc84aa3ccd0643209f8cd71e5851d315" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc84aa3ccd0643209f8cd71e5851d315" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33035 } } }
I20260927 10:53:28.419904 17707 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc84aa3ccd0643209f8cd71e5851d315 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:28.420243 17706 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc84aa3ccd0643209f8cd71e5851d315 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc84aa3ccd0643209f8cd71e5851d315" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc84aa3ccd0643209f8cd71e5851d315" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33035 } } }
I20260927 10:53:28.420327 17706 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc84aa3ccd0643209f8cd71e5851d315 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:28.421178 17714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:28.423909 17714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:28.429364 17714 catalog_manager.cc:1347] Generated new cluster ID: a78bd8842f904437bed6187e3613a729
I20260927 10:53:28.429428 17714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:28.437491 17714 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:28.438637 17714 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:28.444828 17714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc84aa3ccd0643209f8cd71e5851d315: Generated new TSK 0
I20260927 10:53:28.445577 17714 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
I20260927 10:53:28.481009 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:33035
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:28.590878 17726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:28.591164 17726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:28.591209 17726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:28.595355 17726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:28.595414 17726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:28.595474 17726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:53:28.600795 17726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:33035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:28.601876 17726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:28.602926 17726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:28.612735 17731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:28.612735 17734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:28.612758 17732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:28.613104 17726 server_base.cc:1034] running on GCE node
I20260927 10:53:28.613459 17726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:28.614091 17726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:28.615257 17726 hybrid_clock.cc:648] HybridClock initialized: now 1790506408615246 us; error 25 us; skew 500 ppm
Sep 27 10:53:28 dist-test-slave-tz51 krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506408, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:28.617923 17726 init.cc:377] Logged in from keytab as kudu/127.14.68.1@KRBTEST.COM (short username kudu)
I20260927 10:53:28.618847 17726 webserver.cc:460] Webserver started at http://127.14.68.1:44905/ using document root <none> and password file <none>
I20260927 10:53:28.619554 17726 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:28.619621 17726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:28.619867 17726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:28.622023 17726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "9271dd4825944db5b17b214926011672"
format_stamp: "Formatted at 2026-09-27 10:53:28 on dist-test-slave-tz51"
server_key: "5b76c21be4d18850736c179a07130c11"
server_key_iv: "b32980a56311e913747b263a282f8b5c"
server_key_version: "encryptionkey@0"
I20260927 10:53:28.622542 17726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "9271dd4825944db5b17b214926011672"
format_stamp: "Formatted at 2026-09-27 10:53:28 on dist-test-slave-tz51"
server_key: "5b76c21be4d18850736c179a07130c11"
server_key_iv: "b32980a56311e913747b263a282f8b5c"
server_key_version: "encryptionkey@0"
I20260927 10:53:28.626194 17726 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:53:28.628680 17741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:28.629932 17726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:28.630028 17726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "9271dd4825944db5b17b214926011672"
format_stamp: "Formatted at 2026-09-27 10:53:28 on dist-test-slave-tz51"
server_key: "5b76c21be4d18850736c179a07130c11"
server_key_iv: "b32980a56311e913747b263a282f8b5c"
server_key_version: "encryptionkey@0"
I20260927 10:53:28.630113 17726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:28.640657 17726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:28.643575 17726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:28.643795 17726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:28.644321 17726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:28.645061 17726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:28.645113 17726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:28.645150 17726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:28.645174 17726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:28.654122 17726 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:46607
I20260927 10:53:28.654170 17854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:46607 every 8 connection(s)
I20260927 10:53:28.655050 17726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:53:28.657446 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 17726
I20260927 10:53:28.657584 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:53:28.657902 14608 external_mini_cluster.cc:1428] Setting key 715ce831cefba27a59463db02d39263b
Sep 27 10:53:28 dist-test-slave-tz51 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506408, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.2@KRBTEST.COM" created.
I20260927 10:53:28.670660 17703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:28.656911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:39619 (local address 127.14.68.62:33035)
0927 10:53:28.657056 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:28.657060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:28.657983 (+ 923us) server_negotiation.cc:407] Connection header received
0927 10:53:28.659766 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:28.659769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:28.659810 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:28.659880 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:28.661512 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:28.662048 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:28.663911 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:28.664166 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:28.664299 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:28.667121 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:28.667124 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:28.667126 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:28.667148 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:28.668677 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:28.669234 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:28.669237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:28.669238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:28.669310 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:28.669592 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:28.669595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:28.669596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:28.669811 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:28.669896 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:28.670393 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 10:53:28.670535 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:53:28.671592 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:28.657282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:33035 (local address 127.14.68.1:39619)
0927 10:53:28.657821 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:28.657878 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:28.659484 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:28.660096 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:28.660108 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:28.660657 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:28.661368 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:28.661384 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:28.662225 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:28.662231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:28.663787 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:28.663802 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:28.664338 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:28.664342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:28.664524 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:28.665134 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:28.665150 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:28.666953 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:28.668785 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:28.668789 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:28.668801 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:28.669142 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:28.669390 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:28.669392 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:28.669394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:28.669517 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:28.669895 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:28.669899 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:28.670123 (+ 224us) client_negotiation.cc:769] Sending connection context
0927 10:53:28.670358 (+ 235us) client_negotiation.cc:240] Negotiation successful
0927 10:53:28.670727 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":138,"threads_started":1}
I20260927 10:53:28.672978 17855 heartbeater.cc:347] Connected to a master server at 127.14.68.62:33035
I20260927 10:53:28.673228 17855 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:28.673700 17855 heartbeater.cc:511] Master 127.14.68.62:33035 requested a full tablet report, sending...
I20260927 10:53:28.675258 17664 ts_manager.cc:194] Registered new tserver with Master: 9271dd4825944db5b17b214926011672 (127.14.68.1:46607)
I20260927 10:53:28.676443 17664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:39619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
I20260927 10:53:28.685001 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:33035
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:28.794752 17860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:28.794991 17860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:28.795032 17860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:28.799041 17860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:28.799093 17860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:28.799149 17860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:53:28.804312 17860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:33035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:28.805300 17860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:28.806330 17860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:28.817857 17865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:28.817828 17868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:28.817821 17866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:28.825660 17860 server_base.cc:1034] running on GCE node
I20260927 10:53:28.826159 17860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:28.826896 17860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:28.828094 17860 hybrid_clock.cc:648] HybridClock initialized: now 1790506408828047 us; error 51 us; skew 500 ppm
Sep 27 10:53:28 dist-test-slave-tz51 krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506408, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:28.830787 17860 init.cc:377] Logged in from keytab as kudu/127.14.68.2@KRBTEST.COM (short username kudu)
I20260927 10:53:28.831624 17860 webserver.cc:460] Webserver started at http://127.14.68.2:37557/ using document root <none> and password file <none>
I20260927 10:53:28.832240 17860 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:28.832301 17860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:28.832492 17860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:28.834578 17860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "29b0dc707e9348d99968f3e3a94b38b3"
format_stamp: "Formatted at 2026-09-27 10:53:28 on dist-test-slave-tz51"
server_key: "2494670141f83b4c5a7976faf67db611"
server_key_iv: "4c66955e80297d4622b12082e6fb7842"
server_key_version: "encryptionkey@0"
I20260927 10:53:28.835099 17860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "29b0dc707e9348d99968f3e3a94b38b3"
format_stamp: "Formatted at 2026-09-27 10:53:28 on dist-test-slave-tz51"
server_key: "2494670141f83b4c5a7976faf67db611"
server_key_iv: "4c66955e80297d4622b12082e6fb7842"
server_key_version: "encryptionkey@0"
I20260927 10:53:28.838713 17860 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:53:28.841112 17875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:28.842320 17860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:53:28.842411 17860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "29b0dc707e9348d99968f3e3a94b38b3"
format_stamp: "Formatted at 2026-09-27 10:53:28 on dist-test-slave-tz51"
server_key: "2494670141f83b4c5a7976faf67db611"
server_key_iv: "4c66955e80297d4622b12082e6fb7842"
server_key_version: "encryptionkey@0"
I20260927 10:53:28.842492 17860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:28.855963 17860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:28.859009 17860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:28.859176 17860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:28.859684 17860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:28.860484 17860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:28.860531 17860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:28.860569 17860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:28.860585 17860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:28.869360 17860 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:35121
I20260927 10:53:28.869386 17988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:35121 every 8 connection(s)
I20260927 10:53:28.870245 17860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:53:28.870959 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 17860
I20260927 10:53:28.871102 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:53:28.871454 14608 external_mini_cluster.cc:1428] Setting key 0ebe4d2b6bd2116670535cd0dc579c3b
Sep 27 10:53:28 dist-test-slave-tz51 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506408, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.3@KRBTEST.COM" created.
I20260927 10:53:28.885102 17703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:28.872158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:34781 (local address 127.14.68.62:33035)
0927 10:53:28.872298 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:28.872303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:28.873573 (+ 1270us) server_negotiation.cc:407] Connection header received
0927 10:53:28.874944 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:28.874947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:28.874998 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:28.875075 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:28.876388 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:28.877091 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:28.878297 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:28.878749 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:28.878882 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:28.881556 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:28.881563 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:28.881568 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:28.881599 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:28.883044 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:28.883551 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:28.883554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:28.883555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:28.883641 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:28.883969 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:28.883971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:28.883972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:28.884164 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:28.884264 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:28.884727 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 10:53:28.884853 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:53:28.885804 17991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:28.872570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:33035 (local address 127.14.68.2:34781)
0927 10:53:28.873405 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:28.873463 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:28.874724 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:28.875249 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:28.875256 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:28.875635 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:28.876248 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:28.876263 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:28.877219 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:28.877222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:28.878166 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:28.878175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:28.878918 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:28.878922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:28.879065 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:28.879687 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:28.879755 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:28.881388 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:28.883146 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:28.883150 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:28.883162 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:28.883451 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:28.883753 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:28.883756 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:28.883760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:28.883895 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:28.884273 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:28.884278 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:28.884490 (+ 212us) client_negotiation.cc:769] Sending connection context
0927 10:53:28.884636 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 10:53:28.884971 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":151,"threads_started":1}
I20260927 10:53:28.886938 17989 heartbeater.cc:347] Connected to a master server at 127.14.68.62:33035
I20260927 10:53:28.887177 17989 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:28.887634 17989 heartbeater.cc:511] Master 127.14.68.62:33035 requested a full tablet report, sending...
I20260927 10:53:28.888722 17664 ts_manager.cc:194] Registered new tserver with Master: 29b0dc707e9348d99968f3e3a94b38b3 (127.14.68.2:35121)
I20260927 10:53:28.889562 17664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:34781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
I20260927 10:53:28.899770 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:33035
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:29.007146 17994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:29.007371 17994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:29.007411 17994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:29.011355 17994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:29.011404 17994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:29.011458 17994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:53:29.016551 17994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:33035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:29.017491 17994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:29.018507 17994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:29.027982 18002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:29.028017 18000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:29.028031 17994 server_base.cc:1034] running on GCE node
W20260927 10:53:29.028019 17999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:29.028735 17994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:29.029325 17994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:29.030501 17994 hybrid_clock.cc:648] HybridClock initialized: now 1790506409030477 us; error 39 us; skew 500 ppm
Sep 27 10:53:29 dist-test-slave-tz51 krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506409, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:29.033288 17994 init.cc:377] Logged in from keytab as kudu/127.14.68.3@KRBTEST.COM (short username kudu)
I20260927 10:53:29.034090 17994 webserver.cc:460] Webserver started at http://127.14.68.3:45209/ using document root <none> and password file <none>
I20260927 10:53:29.034699 17994 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:29.034794 17994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:29.035009 17994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:29.036957 17994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "867e3d67db5c4e61b1c32cca85a2db85"
format_stamp: "Formatted at 2026-09-27 10:53:29 on dist-test-slave-tz51"
server_key: "f83fab94c857baec28b5903db8d8cd94"
server_key_iv: "5d99aa9e03f71efbd4d4afc6416f6f1b"
server_key_version: "encryptionkey@0"
I20260927 10:53:29.037523 17994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "867e3d67db5c4e61b1c32cca85a2db85"
format_stamp: "Formatted at 2026-09-27 10:53:29 on dist-test-slave-tz51"
server_key: "f83fab94c857baec28b5903db8d8cd94"
server_key_iv: "5d99aa9e03f71efbd4d4afc6416f6f1b"
server_key_version: "encryptionkey@0"
I20260927 10:53:29.041045 17994 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:53:29.043352 18009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:29.044610 17994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:29.044754 17994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "867e3d67db5c4e61b1c32cca85a2db85"
format_stamp: "Formatted at 2026-09-27 10:53:29 on dist-test-slave-tz51"
server_key: "f83fab94c857baec28b5903db8d8cd94"
server_key_iv: "5d99aa9e03f71efbd4d4afc6416f6f1b"
server_key_version: "encryptionkey@0"
I20260927 10:53:29.044866 17994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:29.057181 17994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:29.059834 17994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:29.060014 17994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:29.060537 17994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:29.061308 17994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:29.061385 17994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:29.061460 17994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:29.061511 17994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:29.069908 17994 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:37077
I20260927 10:53:29.069931 18122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:37077 every 8 connection(s)
I20260927 10:53:29.070753 17994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:53:29.075358 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 17994
I20260927 10:53:29.075491 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:53:29.075839 14608 external_mini_cluster.cc:1428] Setting key d21581bee27d90c6029fba1792f2e7be
Sep 27 10:53:29 dist-test-slave-tz51 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506409, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:29.084137 17703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:29.072514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:43647 (local address 127.14.68.62:33035)
0927 10:53:29.072647 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:29.072650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:29.073409 (+ 759us) server_negotiation.cc:407] Connection header received
0927 10:53:29.074722 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:29.074726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:29.074783 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:29.074877 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:29.076349 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:29.077032 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:29.078193 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:29.078417 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:29.078527 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:29.080881 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:29.080885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:29.080887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:29.080921 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:29.082155 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:29.082654 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:29.082664 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:29.082666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:29.082736 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:29.083059 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:29.083064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:29.083066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:29.083241 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:29.083342 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:29.083851 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 10:53:29.083977 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:53:29.084885 18125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:29.072834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:33035 (local address 127.14.68.3:43647)
0927 10:53:29.073250 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:29.073289 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:29.074496 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:29.075036 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:29.075042 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:29.075540 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:29.076186 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:29.076202 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:29.077155 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:29.077159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:29.078067 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:29.078076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:29.078560 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:29.078564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:29.078727 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:29.079300 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:29.079315 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:29.080726 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:29.082269 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:29.082275 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:29.082289 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:29.082535 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:29.082832 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:29.082835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:29.082836 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:29.082965 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:29.083351 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:29.083360 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:29.083584 (+ 224us) client_negotiation.cc:769] Sending connection context
0927 10:53:29.083810 (+ 226us) client_negotiation.cc:240] Negotiation successful
0927 10:53:29.084113 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":105,"threads_started":1}
I20260927 10:53:29.086009 18123 heartbeater.cc:347] Connected to a master server at 127.14.68.62:33035
I20260927 10:53:29.086272 18123 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:29.086715 18123 heartbeater.cc:511] Master 127.14.68.62:33035 requested a full tablet report, sending...
I20260927 10:53:29.087670 17664 ts_manager.cc:194] Registered new tserver with Master: 867e3d67db5c4e61b1c32cca85a2db85 (127.14.68.3:37077)
I20260927 10:53:29.088364 17664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:43647
I20260927 10:53:29.089798 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 10:53:29 dist-test-slave-tz51 krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506409, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 10:53:29 dist-test-slave-tz51 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506409, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:29.113763 17703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:29.105686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60150 (local address 127.14.68.62:33035)
0927 10:53:29.105830 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:29.105835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:29.106016 (+ 181us) server_negotiation.cc:407] Connection header received
0927 10:53:29.106133 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:29.106135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:29.106176 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:29.106255 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:29.106881 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:29.107377 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:29.108239 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:29.108441 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:29.108517 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:29.110712 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:29.110715 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:29.110717 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:29.110741 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:29.112186 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:29.112585 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:29.112587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:29.112588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:29.112659 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:29.112957 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:29.112959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:29.112960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:29.113163 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:29.113276 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:29.113475 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 10:53:29.113564 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 10:53:29.114205 17663 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60150
I20260927 10:53:29.115089 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 17726
I20260927 10:53:29.120743 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 17860
I20260927 10:53:29.125977 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 17994
I20260927 10:53:29.131155 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 17632
2026-09-27T10:53:29Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2041 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:53:29 dist-test-slave-tz51 krb5kdc[18135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:53:29 dist-test-slave-tz51 krb5kdc[18135](info): set up 2 sockets
Sep 27 10:53:29 dist-test-slave-tz51 krb5kdc[18135](info): commencing operation
krb5kdc: starting...
W20260927 10:53:30.171813 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:53:30 dist-test-slave-tz51 krb5kdc[18135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:53:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:53:30.294077 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:35255
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:39823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:35255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:30.403434 18148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:30.403687 18148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:30.403798 18148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:30.407765 18148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:30.407823 18148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:30.407842 18148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:30.407879 18148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:30.407914 18148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:30.413113 18148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:39823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:35255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:35255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:30.414109 18148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:30.415110 18148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:30.423223 18154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:30.423224 18148 server_base.cc:1034] running on GCE node
W20260927 10:53:30.423247 18153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:30.423223 18156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:30.424036 18148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:30.425069 18148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:30.426254 18148 hybrid_clock.cc:648] HybridClock initialized: now 1790506410426226 us; error 44 us; skew 500 ppm
Sep 27 10:53:30 dist-test-slave-tz51 krb5kdc[18135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506410, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:30.428989 18148 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:53:30.429833 18148 webserver.cc:460] Webserver started at http://127.14.68.62:33327/ using document root <none> and password file <none>
I20260927 10:53:30.430325 18148 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:30.430380 18148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:30.430536 18148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:30.432636 18148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "f7230449c9574c3b81efc1aec3631092"
format_stamp: "Formatted at 2026-09-27 10:53:30 on dist-test-slave-tz51"
server_key: "733298def318ffa1038db390f66bc726"
server_key_iv: "f39ee37c42a1aa66fb2af8e44292ec37"
server_key_version: "encryptionkey@0"
I20260927 10:53:30.433215 18148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "f7230449c9574c3b81efc1aec3631092"
format_stamp: "Formatted at 2026-09-27 10:53:30 on dist-test-slave-tz51"
server_key: "733298def318ffa1038db390f66bc726"
server_key_iv: "f39ee37c42a1aa66fb2af8e44292ec37"
server_key_version: "encryptionkey@0"
I20260927 10:53:30.436934 18148 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 10:53:30.439255 18163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:30.440619 18148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:53:30.440770 18148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "f7230449c9574c3b81efc1aec3631092"
format_stamp: "Formatted at 2026-09-27 10:53:30 on dist-test-slave-tz51"
server_key: "733298def318ffa1038db390f66bc726"
server_key_iv: "f39ee37c42a1aa66fb2af8e44292ec37"
server_key_version: "encryptionkey@0"
I20260927 10:53:30.440888 18148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:30.460870 18148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:30.463730 18148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:30.463932 18148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:30.470634 18148 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:35255
I20260927 10:53:30.470659 18215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:35255 every 8 connection(s)
I20260927 10:53:30.471549 18148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:30.474221 18216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:30.479427 18216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7230449c9574c3b81efc1aec3631092: Bootstrap starting.
I20260927 10:53:30.480201 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 18148
I20260927 10:53:30.480330 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:53:30.480639 14608 external_mini_cluster.cc:1428] Setting key 5918b2f4d932d58b29a799badc41ed0c
I20260927 10:53:30.482290 18216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7230449c9574c3b81efc1aec3631092: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:30.483016 18216 log.cc:824] T 00000000000000000000000000000000 P f7230449c9574c3b81efc1aec3631092: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:30.485105 18216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7230449c9574c3b81efc1aec3631092: No bootstrap required, opened a new log
Sep 27 10:53:30 dist-test-slave-tz51 krb5kdc[18135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:30.489509 18216 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7230449c9574c3b81efc1aec3631092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7230449c9574c3b81efc1aec3631092" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 35255 } }
I20260927 10:53:30.489761 18216 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7230449c9574c3b81efc1aec3631092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:30.489846 18216 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7230449c9574c3b81efc1aec3631092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7230449c9574c3b81efc1aec3631092, State: Initialized, Role: FOLLOWER
I20260927 10:53:30.490350 18216 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7230449c9574c3b81efc1aec3631092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7230449c9574c3b81efc1aec3631092" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 35255 } }
I20260927 10:53:30.490517 18216 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7230449c9574c3b81efc1aec3631092 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:30.490590 18216 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7230449c9574c3b81efc1aec3631092 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:30.490674 18216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7230449c9574c3b81efc1aec3631092 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:30.491849 18216 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7230449c9574c3b81efc1aec3631092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7230449c9574c3b81efc1aec3631092" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 35255 } }
I20260927 10:53:30.492264 18216 leader_election.cc:302] T 00000000000000000000000000000000 P f7230449c9574c3b81efc1aec3631092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7230449c9574c3b81efc1aec3631092; no voters:
I20260927 10:53:30.492600 18216 leader_election.cc:288] T 00000000000000000000000000000000 P f7230449c9574c3b81efc1aec3631092 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:30.492667 18221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7230449c9574c3b81efc1aec3631092 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:30.492877 18221 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7230449c9574c3b81efc1aec3631092 [term 1 LEADER]: Becoming Leader. State: Replica: f7230449c9574c3b81efc1aec3631092, State: Running, Role: LEADER
I20260927 10:53:30.493170 18221 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7230449c9574c3b81efc1aec3631092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7230449c9574c3b81efc1aec3631092" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 35255 } }
I20260927 10:53:30.494113 18216 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7230449c9574c3b81efc1aec3631092 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:30.494915 18223 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7230449c9574c3b81efc1aec3631092 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7230449c9574c3b81efc1aec3631092" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7230449c9574c3b81efc1aec3631092" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 35255 } } }
I20260927 10:53:30.495144 18223 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7230449c9574c3b81efc1aec3631092 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:30.495594 18230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:30.495697 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:30.481803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39860 (local address 127.14.68.62:35255)
0927 10:53:30.482208 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:30.482235 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:30.482268 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:53:30.482954 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:30.482969 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:30.483359 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:30.483777 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:30.484602 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:30.485598 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:30.486766 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:30.487233 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:30.487347 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:30.490230 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:30.490236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:30.490251 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:30.490281 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:30.492319 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:30.492789 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:30.492795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:30.492804 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:30.492948 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:30.493261 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:30.493264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:30.493266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:30.493800 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:30.493986 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:30.494281 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 10:53:30.494595 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":126,"threads_started":1}
I20260927 10:53:30.495057 18222 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7230449c9574c3b81efc1aec3631092 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7230449c9574c3b81efc1aec3631092. Latest consensus state: current_term: 1 leader_uuid: "f7230449c9574c3b81efc1aec3631092" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7230449c9574c3b81efc1aec3631092" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 35255 } } }
I20260927 10:53:30.496820 18222 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7230449c9574c3b81efc1aec3631092 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:30.498713 18230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:30.503924 18230 catalog_manager.cc:1347] Generated new cluster ID: 7c4661c107074effb574bbdd22a3a15a
I20260927 10:53:30.503983 18230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:30.512692 18230 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:30.513582 18230 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:30.519656 18230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7230449c9574c3b81efc1aec3631092: Generated new TSK 0
I20260927 10:53:30.520295 18230 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
I20260927 10:53:30.552346 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:35255
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:39823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:30.673947 18242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:30.674201 18242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:30.674243 18242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:30.678398 18242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:30.678463 18242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:30.678561 18242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:53:30.683944 18242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:39823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:35255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:30.685047 18242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:30.686116 18242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:30.695536 18247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:30.695632 18242 server_base.cc:1034] running on GCE node
W20260927 10:53:30.695521 18250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:30.695524 18248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:30.696269 18242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:30.696877 18242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:30.698058 18242 hybrid_clock.cc:648] HybridClock initialized: now 1790506410698048 us; error 29 us; skew 500 ppm
Sep 27 10:53:30 dist-test-slave-tz51 krb5kdc[18135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506410, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:30.700739 18242 init.cc:377] Logged in from keytab as kudu/127.14.68.1@KRBTEST.COM (short username kudu)
I20260927 10:53:30.701537 18242 webserver.cc:460] Webserver started at http://127.14.68.1:41375/ using document root <none> and password file <none>
I20260927 10:53:30.702075 18242 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:30.702164 18242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:30.702405 18242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:30.704454 18242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "f68896154a53467280a05ffb117018a9"
format_stamp: "Formatted at 2026-09-27 10:53:30 on dist-test-slave-tz51"
server_key: "26eda4a1ff765e00fddc126fa875b818"
server_key_iv: "e7a2eb297e56e112b4db7650e7e4ab14"
server_key_version: "encryptionkey@0"
I20260927 10:53:30.704993 18242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "f68896154a53467280a05ffb117018a9"
format_stamp: "Formatted at 2026-09-27 10:53:30 on dist-test-slave-tz51"
server_key: "26eda4a1ff765e00fddc126fa875b818"
server_key_iv: "e7a2eb297e56e112b4db7650e7e4ab14"
server_key_version: "encryptionkey@0"
I20260927 10:53:30.708655 18242 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:53:30.710911 18257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:30.712176 18242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:30.712324 18242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "f68896154a53467280a05ffb117018a9"
format_stamp: "Formatted at 2026-09-27 10:53:30 on dist-test-slave-tz51"
server_key: "26eda4a1ff765e00fddc126fa875b818"
server_key_iv: "e7a2eb297e56e112b4db7650e7e4ab14"
server_key_version: "encryptionkey@0"
I20260927 10:53:30.712433 18242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:30.731997 18242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:30.735363 18242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:30.735549 18242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:30.736168 18242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:30.736985 18242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:30.737053 18242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:30.737123 18242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:30.737176 18242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:30.746457 18242 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:38127
I20260927 10:53:30.746479 18370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:38127 every 8 connection(s)
I20260927 10:53:30.747371 18242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:53:30.749100 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 18242
I20260927 10:53:30.749255 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:53:30.749559 14608 external_mini_cluster.cc:1428] Setting key 0cc78e8bd55c742ad7f63845825f9232
Sep 27 10:53:30 dist-test-slave-tz51 krb5kdc[18135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506410, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.2@KRBTEST.COM" created.
I20260927 10:53:30.763221 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:30.750698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:47535 (local address 127.14.68.62:35255)
0927 10:53:30.750824 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:30.750829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:30.750846 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:53:30.752244 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:30.752250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:30.752306 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:30.752391 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:30.753789 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:30.754451 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:30.756305 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:30.756537 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:30.756617 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:30.759573 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:30.759596 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:30.759601 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:30.759632 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:30.761028 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:30.761571 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:30.761575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:30.761576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:30.761657 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:30.762011 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:30.762016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:30.762018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:30.762252 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:30.762376 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:30.762978 (+ 602us) server_negotiation.cc:299] Negotiation successful
0927 10:53:30.763088 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 10:53:30.764380 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:30.749706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:35255 (local address 127.14.68.1:47535)
0927 10:53:30.750229 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:30.750284 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:30.751992 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:30.752603 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:30.752614 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:30.753127 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:30.753625 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:30.753639 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:30.754596 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:30.754603 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:30.756179 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:30.756192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:30.756703 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:30.756709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:30.756943 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:30.757571 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:30.757593 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:30.759399 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:30.761151 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:30.761156 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:30.761171 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:30.761464 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:30.761769 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:30.761772 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:30.761774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:30.761914 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:30.762365 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:30.762371 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:30.762702 (+ 331us) client_negotiation.cc:769] Sending connection context
0927 10:53:30.762926 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 10:53:30.763308 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":139,"threads_started":1}
I20260927 10:53:30.765704 18371 heartbeater.cc:347] Connected to a master server at 127.14.68.62:35255
I20260927 10:53:30.765960 18371 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:30.766503 18371 heartbeater.cc:511] Master 127.14.68.62:35255 requested a full tablet report, sending...
I20260927 10:53:30.768293 18180 ts_manager.cc:194] Registered new tserver with Master: f68896154a53467280a05ffb117018a9 (127.14.68.1:38127)
I20260927 10:53:30.769945 18180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:47535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
I20260927 10:53:30.779084 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:35255
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:39823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:30.891124 18376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:30.891356 18376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:30.891397 18376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:30.895718 18376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:30.895766 18376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:30.895864 18376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:53:30.900964 18376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:39823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:35255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:30.901952 18376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:30.902931 18376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:30.912077 18381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:30.912122 18382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:30.912122 18384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:30.912173 18376 server_base.cc:1034] running on GCE node
I20260927 10:53:30.912644 18376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:30.913221 18376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:30.914403 18376 hybrid_clock.cc:648] HybridClock initialized: now 1790506410914389 us; error 25 us; skew 500 ppm
Sep 27 10:53:30 dist-test-slave-tz51 krb5kdc[18135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506410, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:30.917196 18376 init.cc:377] Logged in from keytab as kudu/127.14.68.2@KRBTEST.COM (short username kudu)
I20260927 10:53:30.918047 18376 webserver.cc:460] Webserver started at http://127.14.68.2:34059/ using document root <none> and password file <none>
I20260927 10:53:30.918598 18376 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:30.918689 18376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:30.918911 18376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:30.920919 18376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "b4949cefe72d4fc69006df94527ebc61"
format_stamp: "Formatted at 2026-09-27 10:53:30 on dist-test-slave-tz51"
server_key: "65f81bfb7e1e6423c9dc318480ebe54b"
server_key_iv: "185bf20fbf7862c87aec2fb15db3f041"
server_key_version: "encryptionkey@0"
I20260927 10:53:30.921478 18376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "b4949cefe72d4fc69006df94527ebc61"
format_stamp: "Formatted at 2026-09-27 10:53:30 on dist-test-slave-tz51"
server_key: "65f81bfb7e1e6423c9dc318480ebe54b"
server_key_iv: "185bf20fbf7862c87aec2fb15db3f041"
server_key_version: "encryptionkey@0"
I20260927 10:53:30.925037 18376 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:53:30.927362 18391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:30.928576 18376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:30.928714 18376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "b4949cefe72d4fc69006df94527ebc61"
format_stamp: "Formatted at 2026-09-27 10:53:30 on dist-test-slave-tz51"
server_key: "65f81bfb7e1e6423c9dc318480ebe54b"
server_key_iv: "185bf20fbf7862c87aec2fb15db3f041"
server_key_version: "encryptionkey@0"
I20260927 10:53:30.928822 18376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:30.940986 18376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:30.943600 18376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:30.943830 18376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:30.944353 18376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:30.945164 18376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:30.945230 18376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:30.945292 18376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:30.945343 18376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:30.954201 18376 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:44901
I20260927 10:53:30.954221 18504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:44901 every 8 connection(s)
I20260927 10:53:30.955109 18376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
Sep 27 10:53:30 dist-test-slave-tz51 krb5kdc[18135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506410, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:30.965087 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 18376
I20260927 10:53:30.965224 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:53:30.965612 14608 external_mini_cluster.cc:1428] Setting key 4fd231d154344e09e3f61baeaac1cf61
I20260927 10:53:30.968695 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:30.956803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:51127 (local address 127.14.68.62:35255)
0927 10:53:30.956963 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:30.956968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:30.957794 (+ 826us) server_negotiation.cc:407] Connection header received
0927 10:53:30.959238 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:30.959241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:30.959288 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:30.959353 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:30.960553 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:30.961072 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:30.962240 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:30.962446 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:30.962518 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:30.964960 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:30.964966 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:30.964970 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:30.964996 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:30.966591 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:30.967186 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:30.967191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:30.967192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:30.967278 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:30.967609 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:30.967615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:30.967616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:30.967854 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:30.967960 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:30.968424 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 10:53:30.968535 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 10:53:30.969462 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:30.957207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:35255 (local address 127.14.68.2:51127)
0927 10:53:30.957650 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:30.957690 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:30.958881 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:30.959513 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:30.959520 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:30.959951 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:30.960417 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:30.960430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:30.961296 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:30.961299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:30.962135 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:30.962143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:30.962565 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:30.962567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:30.962705 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:30.963302 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:30.963317 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:30.964802 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:30.966707 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:30.966713 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:30.966743 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:30.967073 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:30.967376 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:30.967379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:30.967381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:30.967517 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:30.967959 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:30.967965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:30.968197 (+ 232us) client_negotiation.cc:769] Sending connection context
0927 10:53:30.968383 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 10:53:30.968676 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":104,"threads_started":1}
I20260927 10:53:30.970554 18505 heartbeater.cc:347] Connected to a master server at 127.14.68.62:35255
I20260927 10:53:30.970796 18505 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:30.971271 18505 heartbeater.cc:511] Master 127.14.68.62:35255 requested a full tablet report, sending...
I20260927 10:53:30.972317 18180 ts_manager.cc:194] Registered new tserver with Master: b4949cefe72d4fc69006df94527ebc61 (127.14.68.2:44901)
I20260927 10:53:30.973151 18180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:51127
WARNING: no policy specified for kudu/127.14.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
I20260927 10:53:30.992147 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:35255
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:39823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:31.099998 18510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:31.100270 18510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:31.100342 18510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:31.104278 18510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:31.104362 18510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:31.104450 18510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:53:31.109480 18510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:39823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:35255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:31.110531 18510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:31.111572 18510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:31.121075 18518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:31.121104 18515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:31.121129 18510 server_base.cc:1034] running on GCE node
W20260927 10:53:31.121076 18516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:31.121660 18510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:31.122263 18510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:31.123414 18510 hybrid_clock.cc:648] HybridClock initialized: now 1790506411123385 us; error 43 us; skew 500 ppm
Sep 27 10:53:31 dist-test-slave-tz51 krb5kdc[18135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506411, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:31.126206 18510 init.cc:377] Logged in from keytab as kudu/127.14.68.3@KRBTEST.COM (short username kudu)
I20260927 10:53:31.127012 18510 webserver.cc:460] Webserver started at http://127.14.68.3:42015/ using document root <none> and password file <none>
I20260927 10:53:31.127612 18510 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:31.127676 18510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:31.127907 18510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:31.129783 18510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "04c0eebfc7ca407891ad54c22df3fda6"
format_stamp: "Formatted at 2026-09-27 10:53:31 on dist-test-slave-tz51"
server_key: "93bd67f91f1c73d9872d6ea8f5e78e66"
server_key_iv: "273054527af67243d841e30e00356856"
server_key_version: "encryptionkey@0"
I20260927 10:53:31.130257 18510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "04c0eebfc7ca407891ad54c22df3fda6"
format_stamp: "Formatted at 2026-09-27 10:53:31 on dist-test-slave-tz51"
server_key: "93bd67f91f1c73d9872d6ea8f5e78e66"
server_key_iv: "273054527af67243d841e30e00356856"
server_key_version: "encryptionkey@0"
I20260927 10:53:31.133929 18510 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 10:53:31.136251 18525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:31.137535 18510 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:31.137655 18510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "04c0eebfc7ca407891ad54c22df3fda6"
format_stamp: "Formatted at 2026-09-27 10:53:31 on dist-test-slave-tz51"
server_key: "93bd67f91f1c73d9872d6ea8f5e78e66"
server_key_iv: "273054527af67243d841e30e00356856"
server_key_version: "encryptionkey@0"
I20260927 10:53:31.137758 18510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:31.155164 18510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:31.158293 18510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:31.158478 18510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:31.159024 18510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:31.159878 18510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:31.159924 18510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:31.159984 18510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:31.160022 18510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:31.168676 18510 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:45591
I20260927 10:53:31.168751 18638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:45591 every 8 connection(s)
I20260927 10:53:31.169525 18510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:53:31.178062 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 18510
I20260927 10:53:31.178216 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:53:31.178479 14608 external_mini_cluster.cc:1428] Setting key b9974dd3353659f3ad074482dfcda44c
Sep 27 10:53:31 dist-test-slave-tz51 krb5kdc[18135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506411, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:31.182456 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:31.171378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:41701 (local address 127.14.68.62:35255)
0927 10:53:31.171501 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:31.171505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:31.172295 (+ 790us) server_negotiation.cc:407] Connection header received
0927 10:53:31.173564 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:31.173567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:31.173622 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:31.173720 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:31.174772 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:31.175285 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:31.176487 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:31.176698 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:31.176807 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:31.179285 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:31.179291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:31.179297 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:31.179329 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:31.180879 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:31.181306 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:31.181308 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:31.181309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:31.181378 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:31.181642 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:31.181644 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:31.181645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:31.181796 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:31.181872 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:31.182262 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 10:53:31.182348 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:53:31.183243 18641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:31.171733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:35255 (local address 127.14.68.3:41701)
0927 10:53:31.172156 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:31.172194 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:31.173370 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:31.173888 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:31.173894 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:31.174196 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:31.174628 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:31.174640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:31.175405 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:31.175409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:31.176353 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:31.176362 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:31.176840 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:31.176844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:31.176998 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:31.177521 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:31.177536 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:31.179065 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:31.180975 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:31.180979 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:31.180990 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:31.181223 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:31.181451 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:31.181453 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:31.181454 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:31.181570 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:31.181882 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:31.181886 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:31.182080 (+ 194us) client_negotiation.cc:769] Sending connection context
0927 10:53:31.182218 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 10:53:31.182433 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":114,"threads_started":1}
I20260927 10:53:31.184258 18639 heartbeater.cc:347] Connected to a master server at 127.14.68.62:35255
I20260927 10:53:31.184478 18639 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:31.184921 18639 heartbeater.cc:511] Master 127.14.68.62:35255 requested a full tablet report, sending...
I20260927 10:53:31.185832 18180 ts_manager.cc:194] Registered new tserver with Master: 04c0eebfc7ca407891ad54c22df3fda6 (127.14.68.3:45591)
I20260927 10:53:31.186414 18180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:41701
I20260927 10:53:31.191967 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 10:53:31 dist-test-slave-tz51 krb5kdc[18135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:53:31 dist-test-slave-tz51 krb5kdc[18135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.1@KRBTEST.COM
I20260927 10:53:31.216954 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:31.207517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42304 (local address 127.14.68.1:38127)
0927 10:53:31.207906 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:31.207921 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:31.207953 (+ 32us) server_negotiation.cc:407] Connection header received
0927 10:53:31.207998 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:31.208003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:31.208175 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:31.208335 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:31.209023 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:31.209810 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:31.210677 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:31.211058 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:31.211182 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:31.213529 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:31.213538 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:31.213551 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:31.213576 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:31.215031 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:31.215468 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:31.215474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:31.215475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:31.215553 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:31.215960 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:31.215965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:31.215967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:31.216210 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:31.216455 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:31.216622 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 10:53:31.216775 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":128,"threads_started":1}
Sep 27 10:53:31 dist-test-slave-tz51 krb5kdc[18135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:31.227116 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:31.219790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39872 (local address 127.14.68.62:35255)
0927 10:53:31.219965 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:31.219969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:31.220003 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:53:31.220118 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:31.220140 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:31.220183 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:31.220249 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:31.220765 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:31.221274 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:31.221982 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:31.222203 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:31.222280 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:31.224442 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:31.224444 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:31.224446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:31.224468 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:31.225662 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:31.226013 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:31.226016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:31.226017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:31.226098 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:31.226343 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:31.226345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:31.226347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:31.226541 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:31.226639 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:31.226822 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 10:53:31.226910 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 10:53:31.227527 18180 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39872
I20260927 10:53:31.228159 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 18242
I20260927 10:53:31.233711 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 18376
I20260927 10:53:31.240999 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 18510
I20260927 10:53:31.246101 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 18148
2026-09-27T10:53:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2115 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:53:31 dist-test-slave-tz51 krb5kdc[18651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:53:31 dist-test-slave-tz51 krb5kdc[18651](info): set up 2 sockets
Sep 27 10:53:31 dist-test-slave-tz51 krb5kdc[18651](info): commencing operation
krb5kdc: starting...
W20260927 10:53:32.275799 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:53:32 dist-test-slave-tz51 krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:53:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:53:32.395524 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:41189
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:40431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:41189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:32.503352 18664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:32.503655 18664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:32.503777 18664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:32.507795 18664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:32.507886 18664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:32.507927 18664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:32.507963 18664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:32.507995 18664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:32.513166 18664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:40431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:41189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:41189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:32.514216 18664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:32.515244 18664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:53:32.523504 18664 server_base.cc:1034] running on GCE node
W20260927 10:53:32.523459 18672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:32.523484 18670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:32.523468 18669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:32.524262 18664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:32.525302 18664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:32.526494 18664 hybrid_clock.cc:648] HybridClock initialized: now 1790506412526462 us; error 42 us; skew 500 ppm
Sep 27 10:53:32 dist-test-slave-tz51 krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506412, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:32.535233 18664 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:53:32.536052 18664 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:32.536170 18664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:32.536386 18664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:32.538411 18664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "4ad75c1148e144fcbbef53b8b81b3dac"
format_stamp: "Formatted at 2026-09-27 10:53:32 on dist-test-slave-tz51"
server_key: "29231e862a587751834d7b567aa46d0f"
server_key_iv: "b51e5d6fc0c428fbc4db223612f3cca2"
server_key_version: "encryptionkey@0"
I20260927 10:53:32.539022 18664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "4ad75c1148e144fcbbef53b8b81b3dac"
format_stamp: "Formatted at 2026-09-27 10:53:32 on dist-test-slave-tz51"
server_key: "29231e862a587751834d7b567aa46d0f"
server_key_iv: "b51e5d6fc0c428fbc4db223612f3cca2"
server_key_version: "encryptionkey@0"
I20260927 10:53:32.542532 18664 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 10:53:32.544800 18678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:32.546001 18664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:53:32.546150 18664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "4ad75c1148e144fcbbef53b8b81b3dac"
format_stamp: "Formatted at 2026-09-27 10:53:32 on dist-test-slave-tz51"
server_key: "29231e862a587751834d7b567aa46d0f"
server_key_iv: "b51e5d6fc0c428fbc4db223612f3cca2"
server_key_version: "encryptionkey@0"
I20260927 10:53:32.546257 18664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:32.556921 18664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:32.560039 18664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:32.560230 18664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:32.566828 18664 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:41189
I20260927 10:53:32.567085 18730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:41189 every 8 connection(s)
I20260927 10:53:32.567463 18664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:32.570210 18731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:32.571223 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 18664
I20260927 10:53:32.571321 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:53:32.571612 14608 external_mini_cluster.cc:1428] Setting key 030934ac00725d7ba967517c508e4725
I20260927 10:53:32.576349 18731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac: Bootstrap starting.
I20260927 10:53:32.579034 18731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:32.580037 18731 log.cc:824] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:53:32 dist-test-slave-tz51 krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:32.582041 18731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac: No bootstrap required, opened a new log
I20260927 10:53:32.585665 18731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ad75c1148e144fcbbef53b8b81b3dac" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41189 } }
I20260927 10:53:32.585850 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:32.572991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46538 (local address 127.14.68.62:41189)
0927 10:53:32.573381 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:32.573396 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:32.573430 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:53:32.574036 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:32.574052 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:32.574364 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:32.574636 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:32.575520 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:32.576578 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:32.577757 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:32.578227 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:32.578349 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:32.581331 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:32.581336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:32.581350 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:32.581375 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:32.583063 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:32.583520 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:32.583530 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:32.583535 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:32.583651 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:32.583965 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:32.583969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:32.583971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:32.584354 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:32.584548 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:32.584838 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 10:53:32.585117 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":119,"threads_started":1}
I20260927 10:53:32.585928 18731 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:32.586001 18731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ad75c1148e144fcbbef53b8b81b3dac, State: Initialized, Role: FOLLOWER
I20260927 10:53:32.586512 18731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ad75c1148e144fcbbef53b8b81b3dac" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41189 } }
I20260927 10:53:32.586664 18731 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:32.586736 18731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:32.586830 18731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:32.588011 18731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ad75c1148e144fcbbef53b8b81b3dac" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41189 } }
I20260927 10:53:32.588370 18731 leader_election.cc:302] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ad75c1148e144fcbbef53b8b81b3dac; no voters:
I20260927 10:53:32.588685 18731 leader_election.cc:288] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:32.588812 18736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:32.589067 18736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac [term 1 LEADER]: Becoming Leader. State: Replica: 4ad75c1148e144fcbbef53b8b81b3dac, State: Running, Role: LEADER
I20260927 10:53:32.589385 18736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ad75c1148e144fcbbef53b8b81b3dac" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41189 } }
I20260927 10:53:32.589936 18731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:32.591138 18738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ad75c1148e144fcbbef53b8b81b3dac. Latest consensus state: current_term: 1 leader_uuid: "4ad75c1148e144fcbbef53b8b81b3dac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ad75c1148e144fcbbef53b8b81b3dac" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41189 } } }
I20260927 10:53:32.591166 18737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ad75c1148e144fcbbef53b8b81b3dac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ad75c1148e144fcbbef53b8b81b3dac" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41189 } } }
I20260927 10:53:32.591279 18738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:32.591279 18737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:32.591753 18745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:32.594558 18745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:32.599898 18745 catalog_manager.cc:1347] Generated new cluster ID: a57a222cecf842e4a133f97320c6fa6c
I20260927 10:53:32.600010 18745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:32.611227 18745 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:32.612320 18745 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:32.621227 18745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac: Generated new TSK 0
I20260927 10:53:32.621941 18745 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
I20260927 10:53:32.655755 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:41189
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:40431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:32.764353 18757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:32.764585 18757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:32.764626 18757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:32.768568 18757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:32.768615 18757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:32.768671 18757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:53:32.773643 18757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:40431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:41189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:32.774503 18757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:32.775430 18757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:32.784412 18763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:32.784427 18765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:32.784462 18757 server_base.cc:1034] running on GCE node
W20260927 10:53:32.784406 18762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:32.784973 18757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:32.785578 18757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:32.786763 18757 hybrid_clock.cc:648] HybridClock initialized: now 1790506412786745 us; error 31 us; skew 500 ppm
Sep 27 10:53:32 dist-test-slave-tz51 krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506412, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:32.789412 18757 init.cc:377] Logged in from keytab as kudu/127.14.68.1@KRBTEST.COM (short username kudu)
I20260927 10:53:32.790153 18757 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:32.790247 18757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:32.790484 18757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:32.792606 18757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "f97366d0c39743b3b0ae64e54d652a45"
format_stamp: "Formatted at 2026-09-27 10:53:32 on dist-test-slave-tz51"
server_key: "4e3b6ba5efb46ff0d735170d7f191e5f"
server_key_iv: "fe76e1082ebdc5929d298eb3598a1be3"
server_key_version: "encryptionkey@0"
I20260927 10:53:32.793171 18757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "f97366d0c39743b3b0ae64e54d652a45"
format_stamp: "Formatted at 2026-09-27 10:53:32 on dist-test-slave-tz51"
server_key: "4e3b6ba5efb46ff0d735170d7f191e5f"
server_key_iv: "fe76e1082ebdc5929d298eb3598a1be3"
server_key_version: "encryptionkey@0"
I20260927 10:53:32.796650 18757 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:53:32.799626 18771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:32.801240 18757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:53:32.801354 18757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "f97366d0c39743b3b0ae64e54d652a45"
format_stamp: "Formatted at 2026-09-27 10:53:32 on dist-test-slave-tz51"
server_key: "4e3b6ba5efb46ff0d735170d7f191e5f"
server_key_iv: "fe76e1082ebdc5929d298eb3598a1be3"
server_key_version: "encryptionkey@0"
I20260927 10:53:32.801462 18757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:32.842654 18757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:32.845664 18757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:32.845870 18757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:32.846482 18757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:32.847306 18757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:32.847350 18757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:32.847419 18757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:32.847450 18757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:32.856288 18757 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:46199
I20260927 10:53:32.856303 18884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:46199 every 8 connection(s)
I20260927 10:53:32.856877 18757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:53:32.861949 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 18757
I20260927 10:53:32.862080 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:53:32.862362 14608 external_mini_cluster.cc:1428] Setting key 6411418fc59e45dafd1f3d2755333475
Sep 27 10:53:32 dist-test-slave-tz51 krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506412, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:32.870921 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:32.858523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:55669 (local address 127.14.68.62:41189)
0927 10:53:32.858651 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:32.858655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:32.859469 (+ 814us) server_negotiation.cc:407] Connection header received
0927 10:53:32.860786 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:32.860790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:32.860848 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:32.860923 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:32.862033 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:32.862575 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:32.863952 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:32.864207 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:32.864319 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:32.867025 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:32.867033 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:32.867039 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:32.867073 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:32.868891 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:32.869370 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:32.869376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:32.869379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:32.869477 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:32.869806 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:32.869812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:32.869813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:32.870056 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:32.870164 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:32.870609 (+ 445us) server_negotiation.cc:299] Negotiation successful
0927 10:53:32.870735 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:53:32.871652 18887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:32.858902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:41189 (local address 127.14.68.1:55669)
0927 10:53:32.859331 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:32.859375 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:32.860570 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:32.861083 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:32.861089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:32.861425 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:32.861882 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:32.861895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:32.862706 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:32.862710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:32.863838 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:32.863848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:32.864352 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:32.864356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:32.864527 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:32.865169 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:32.865194 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:32.866854 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:32.869006 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:32.869010 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:32.869021 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:32.869270 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:32.869577 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:32.869580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:32.869581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:32.869703 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:32.870161 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:32.870165 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:32.870377 (+ 212us) client_negotiation.cc:769] Sending connection context
0927 10:53:32.870522 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 10:53:32.870764 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.14.68.2@KRBTEST.COM; defaulting to no policy
I20260927 10:53:32.872886 18885 heartbeater.cc:347] Connected to a master server at 127.14.68.62:41189
I20260927 10:53:32.873127 18885 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:32.873535 18885 heartbeater.cc:511] Master 127.14.68.62:41189 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.2@KRBTEST.COM" created.
I20260927 10:53:32.875078 18695 ts_manager.cc:194] Registered new tserver with Master: f97366d0c39743b3b0ae64e54d652a45 (127.14.68.1:46199)
I20260927 10:53:32.876572 18695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:55669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
I20260927 10:53:32.890724 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:41189
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:40431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:32.998294 18890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:32.998586 18890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:32.998667 18890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:33.002645 18890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:33.002728 18890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:33.002820 18890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:53:33.008376 18890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:40431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:41189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:33.009500 18890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:33.010545 18890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:53:33.019969 18890 server_base.cc:1034] running on GCE node
W20260927 10:53:33.019903 18895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:33.019906 18898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:33.019901 18896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:33.020521 18890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:33.021148 18890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:33.022306 18890 hybrid_clock.cc:648] HybridClock initialized: now 1790506413022289 us; error 29 us; skew 500 ppm
Sep 27 10:53:33 dist-test-slave-tz51 krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506413, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:33.025098 18890 init.cc:377] Logged in from keytab as kudu/127.14.68.2@KRBTEST.COM (short username kudu)
I20260927 10:53:33.025861 18890 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:33.025952 18890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:33.026192 18890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:33.028277 18890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "4a6080c03f914d8d9a8210c01a6dbab6"
format_stamp: "Formatted at 2026-09-27 10:53:33 on dist-test-slave-tz51"
server_key: "303cda8a4dcccfe7721e8cf5b6475481"
server_key_iv: "de665412c8fc2f1650591e63b9dcefcd"
server_key_version: "encryptionkey@0"
I20260927 10:53:33.028858 18890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "4a6080c03f914d8d9a8210c01a6dbab6"
format_stamp: "Formatted at 2026-09-27 10:53:33 on dist-test-slave-tz51"
server_key: "303cda8a4dcccfe7721e8cf5b6475481"
server_key_iv: "de665412c8fc2f1650591e63b9dcefcd"
server_key_version: "encryptionkey@0"
I20260927 10:53:33.032634 18890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:53:33.035033 18904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:33.036304 18890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:33.036440 18890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "4a6080c03f914d8d9a8210c01a6dbab6"
format_stamp: "Formatted at 2026-09-27 10:53:33 on dist-test-slave-tz51"
server_key: "303cda8a4dcccfe7721e8cf5b6475481"
server_key_iv: "de665412c8fc2f1650591e63b9dcefcd"
server_key_version: "encryptionkey@0"
I20260927 10:53:33.036551 18890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:33.081285 18890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:33.084408 18890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:33.084616 18890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:33.085309 18890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:33.086198 18890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:33.086244 18890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:33.086283 18890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:33.086298 18890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:33.096081 18890 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:37139
I20260927 10:53:33.096105 19017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:37139 every 8 connection(s)
I20260927 10:53:33.096665 18890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
Sep 27 10:53:33 dist-test-slave-tz51 krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506413, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:33.106907 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 18890
I20260927 10:53:33.107051 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:53:33.107394 14608 external_mini_cluster.cc:1428] Setting key 1a16f0a067e6e5cd5834a6df9c6d7eab
I20260927 10:53:33.110416 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:33.098319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:44103 (local address 127.14.68.62:41189)
0927 10:53:33.098456 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:33.098460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:33.099307 (+ 847us) server_negotiation.cc:407] Connection header received
0927 10:53:33.100572 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:33.100575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:33.100631 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:33.100736 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:33.101858 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.102409 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.103544 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.103830 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.103961 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:33.106469 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:33.106473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:33.106476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:33.106505 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:33.108253 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:33.108861 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:33.108863 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:33.108864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:33.108941 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:33.109317 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:33.109319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:33.109320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:33.109477 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:33.109558 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:33.110189 (+ 631us) server_negotiation.cc:299] Negotiation successful
0927 10:53:33.110296 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:53:33.111387 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:33.098725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:41189 (local address 127.14.68.2:44103)
0927 10:53:33.099157 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:33.099204 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:33.100365 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:33.100900 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:33.100907 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:33.101237 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:33.101709 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:33.101722 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.102532 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.102536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:33.103418 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:33.103426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.103975 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.103979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:33.104160 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:33.104759 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:33.104776 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:33.106299 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:33.108360 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:33.108366 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:33.108380 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:33.108711 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:33.109036 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:33.109039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:33.109041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:33.109224 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:33.109594 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:33.109603 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:33.109931 (+ 328us) client_negotiation.cc:769] Sending connection context
0927 10:53:33.110145 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 10:53:33.110507 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":108,"threads_started":1}
I20260927 10:53:33.112649 19018 heartbeater.cc:347] Connected to a master server at 127.14.68.62:41189
I20260927 10:53:33.112905 19018 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:33.113358 19018 heartbeater.cc:511] Master 127.14.68.62:41189 requested a full tablet report, sending...
I20260927 10:53:33.114454 18695 ts_manager.cc:194] Registered new tserver with Master: 4a6080c03f914d8d9a8210c01a6dbab6 (127.14.68.2:37139)
I20260927 10:53:33.115021 18695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:44103
WARNING: no policy specified for kudu/127.14.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
I20260927 10:53:33.135562 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:41189
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:40431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:33.243952 19023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:33.244199 19023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:33.244243 19023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:33.248276 19023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:33.248328 19023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:33.248430 19023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:53:33.253695 19023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:40431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:41189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:33.254655 19023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:33.255606 19023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:33.264509 19028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:33.264487 19029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:33.264482 19031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:33.264909 19023 server_base.cc:1034] running on GCE node
I20260927 10:53:33.265288 19023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:33.265918 19023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:33.267094 19023 hybrid_clock.cc:648] HybridClock initialized: now 1790506413267082 us; error 32 us; skew 500 ppm
Sep 27 10:53:33 dist-test-slave-tz51 krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506413, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:33.269800 19023 init.cc:377] Logged in from keytab as kudu/127.14.68.3@KRBTEST.COM (short username kudu)
I20260927 10:53:33.270466 19023 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:33.270560 19023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:33.270802 19023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:33.272879 19023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "7e494085174f441a89e020fdd2653195"
format_stamp: "Formatted at 2026-09-27 10:53:33 on dist-test-slave-tz51"
server_key: "ca5ba1225b1e55eb052a031c7a38890a"
server_key_iv: "370b70111ab746ff0d2b5ea9308be12e"
server_key_version: "encryptionkey@0"
I20260927 10:53:33.273416 19023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "7e494085174f441a89e020fdd2653195"
format_stamp: "Formatted at 2026-09-27 10:53:33 on dist-test-slave-tz51"
server_key: "ca5ba1225b1e55eb052a031c7a38890a"
server_key_iv: "370b70111ab746ff0d2b5ea9308be12e"
server_key_version: "encryptionkey@0"
I20260927 10:53:33.277000 19023 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:53:33.279294 19037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:33.280541 19023 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:33.280694 19023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "7e494085174f441a89e020fdd2653195"
format_stamp: "Formatted at 2026-09-27 10:53:33 on dist-test-slave-tz51"
server_key: "ca5ba1225b1e55eb052a031c7a38890a"
server_key_iv: "370b70111ab746ff0d2b5ea9308be12e"
server_key_version: "encryptionkey@0"
I20260927 10:53:33.280817 19023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:33.295015 19023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:33.297714 19023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:33.297928 19023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:33.298472 19023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:33.299278 19023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:33.299371 19023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:33.299450 19023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:33.299515 19023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:33.308256 19023 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:46777
I20260927 10:53:33.308276 19150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:46777 every 8 connection(s)
I20260927 10:53:33.308844 19023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:53:33.311450 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 19023
I20260927 10:53:33.311586 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:53:33.311913 14608 external_mini_cluster.cc:1428] Setting key e0718b0871347fc12f0029365012a320
Sep 27 10:53:33 dist-test-slave-tz51 krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506413, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:33.321825 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:33.310537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:47035 (local address 127.14.68.62:41189)
0927 10:53:33.310711 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:33.310717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:33.311521 (+ 804us) server_negotiation.cc:407] Connection header received
0927 10:53:33.313145 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:33.313151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:33.313209 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:33.313281 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:33.314565 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.315130 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.316296 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.316514 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.316602 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:33.318807 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:33.318810 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:33.318812 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:33.318834 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:33.320110 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:33.320564 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:33.320567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:33.320569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:33.320657 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:33.320973 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:33.320976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:33.320977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:33.321136 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:33.321210 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:33.321615 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 10:53:33.321705 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 10:53:33.322559 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:33.310926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:41189 (local address 127.14.68.3:47035)
0927 10:53:33.311375 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:33.311416 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:33.312881 (+ 1465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:33.313481 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:33.313488 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:33.313879 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:33.314437 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:33.314451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.315244 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.315247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:33.316186 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:33.316195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.316643 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.316646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:33.316780 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:33.317298 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:33.317313 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:33.318666 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:33.320207 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:33.320211 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:33.320222 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:33.320463 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:33.320758 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:33.320760 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:33.320761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:33.320893 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:33.321219 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:33.321223 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:33.321422 (+ 199us) client_negotiation.cc:769] Sending connection context
0927 10:53:33.321569 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 10:53:33.321797 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":121,"threads_started":1}
I20260927 10:53:33.323549 19151 heartbeater.cc:347] Connected to a master server at 127.14.68.62:41189
I20260927 10:53:33.323799 19151 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:33.324296 19151 heartbeater.cc:511] Master 127.14.68.62:41189 requested a full tablet report, sending...
I20260927 10:53:33.325439 18695 ts_manager.cc:194] Registered new tserver with Master: 7e494085174f441a89e020fdd2653195 (127.14.68.3:46777)
I20260927 10:53:33.326061 18695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:47035
I20260927 10:53:33.337354 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:53:33.346649 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:33.339461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46552 (local address 127.14.68.62:41189)
0927 10:53:33.339619 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:33.339622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:33.339832 (+ 210us) server_negotiation.cc:407] Connection header received
0927 10:53:33.339923 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:33.339925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:33.339964 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:33.340031 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:33.340828 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.341334 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.342145 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.342346 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.342439 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:33.343813 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:33.343816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:33.343818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:33.343839 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:33.345231 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:33.345597 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:33.345599 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:33.345601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:33.345668 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:33.345886 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:33.345888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:33.345889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:33.346096 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:33.346198 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:33.346441 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 10:53:33.346535 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 10:53:33.349632 18695 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:53:33.352792 18695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:33.367213 19165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:33.361923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:46777 (local address 127.0.0.1:34288)
0927 10:53:33.362545 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:33.362558 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:33.362658 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:33.363266 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:33.363271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:33.363290 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:33.363406 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:33.363412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.364837 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.364840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:33.365742 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:33.365750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.366645 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.366649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:33.366774 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:33.366815 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 10:53:33.366928 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 10:53:33.367040 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260927 10:53:33.367233 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:33.362123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34288 (local address 127.14.68.3:46777)
0927 10:53:33.362476 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:33.362489 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:33.362634 (+ 145us) server_negotiation.cc:407] Connection header received
0927 10:53:33.362776 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:33.362780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:33.362956 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:33.363100 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:33.363536 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.364669 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.365865 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.366524 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.366646 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:33.366784 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:33.366970 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 10:53:33.367065 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":179,"threads_started":1}
I20260927 10:53:33.367661 19162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:33.361156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:46199 (local address 127.0.0.1:50438)
0927 10:53:33.361751 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:33.361803 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:33.361922 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:33.362986 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:33.362992 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:33.363018 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:33.363161 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:33.363169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.364667 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.364671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:33.366239 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:33.366248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.367279 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.367285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:33.367404 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:33.367426 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:53:33.367471 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 10:53:33.367527 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":257,"thread_start_us":73,"threads_started":1}
I20260927 10:53:33.367869 19163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:33.361238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:37139 (local address 127.0.0.1:46760)
0927 10:53:33.362165 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:33.362179 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:33.362254 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:33.363207 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:33.363210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:33.363227 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:33.363323 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:33.363330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.364427 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.364432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:33.366238 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:33.366248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.367471 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.367474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:33.367547 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:33.367564 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 10:53:33.367653 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 10:53:33.367745 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":72,"thread_start_us":94,"threads_started":1}
I20260927 10:53:33.367980 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:33.361961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46760 (local address 127.14.68.2:37139)
0927 10:53:33.362735 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:33.362746 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:33.362766 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:53:33.362834 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:33.362840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:33.362998 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:33.363125 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:33.363448 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.364304 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.366373 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.367143 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.367263 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:33.367407 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:33.367687 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 10:53:33.367822 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":115,"threads_started":1}
I20260927 10:53:33.368599 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:33.361298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.14.68.1:46199)
0927 10:53:33.362485 (+ 1187us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:33.362491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:33.362515 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:53:33.362596 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:33.362603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:33.362827 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:33.362981 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:33.363300 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.364530 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.366379 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.367130 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.368182 (+ 1052us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:33.368325 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:33.368370 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 10:53:33.368453 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1076,"thread_start_us":84,"threads_started":1}
I20260927 10:53:33.370627 19085 tablet_service.cc:1467] Processing CreateTablet for tablet 9b02cbfd9e734c30a1aafd997535569b (DEFAULT_TABLE table=test-table [id=23bf90c7b93a4108a0c88b03fb686775]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:33.371606 19085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b02cbfd9e734c30a1aafd997535569b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:33.371299 18952 tablet_service.cc:1467] Processing CreateTablet for tablet 9b02cbfd9e734c30a1aafd997535569b (DEFAULT_TABLE table=test-table [id=23bf90c7b93a4108a0c88b03fb686775]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:33.371464 18819 tablet_service.cc:1467] Processing CreateTablet for tablet 9b02cbfd9e734c30a1aafd997535569b (DEFAULT_TABLE table=test-table [id=23bf90c7b93a4108a0c88b03fb686775]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:33.372367 18952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b02cbfd9e734c30a1aafd997535569b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:33.372367 18819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b02cbfd9e734c30a1aafd997535569b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:33.378432 19168 tablet_bootstrap.cc:492] T 9b02cbfd9e734c30a1aafd997535569b P 7e494085174f441a89e020fdd2653195: Bootstrap starting.
I20260927 10:53:33.380029 19169 tablet_bootstrap.cc:492] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45: Bootstrap starting.
I20260927 10:53:33.380666 19168 tablet_bootstrap.cc:654] T 9b02cbfd9e734c30a1aafd997535569b P 7e494085174f441a89e020fdd2653195: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:33.381742 19168 log.cc:824] T 9b02cbfd9e734c30a1aafd997535569b P 7e494085174f441a89e020fdd2653195: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:33.381870 19169 tablet_bootstrap.cc:654] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:33.382437 19170 tablet_bootstrap.cc:492] T 9b02cbfd9e734c30a1aafd997535569b P 4a6080c03f914d8d9a8210c01a6dbab6: Bootstrap starting.
I20260927 10:53:33.382712 19169 log.cc:824] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:33.383875 19168 tablet_bootstrap.cc:492] T 9b02cbfd9e734c30a1aafd997535569b P 7e494085174f441a89e020fdd2653195: No bootstrap required, opened a new log
I20260927 10:53:33.384091 19168 ts_tablet_manager.cc:1397] T 9b02cbfd9e734c30a1aafd997535569b P 7e494085174f441a89e020fdd2653195: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:53:33.384735 19169 tablet_bootstrap.cc:492] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45: No bootstrap required, opened a new log
I20260927 10:53:33.384829 19170 tablet_bootstrap.cc:654] T 9b02cbfd9e734c30a1aafd997535569b P 4a6080c03f914d8d9a8210c01a6dbab6: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:33.384917 19169 ts_tablet_manager.cc:1397] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:53:33.385689 19170 log.cc:824] T 9b02cbfd9e734c30a1aafd997535569b P 4a6080c03f914d8d9a8210c01a6dbab6: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:33.387377 19168 raft_consensus.cc:348] T 9b02cbfd9e734c30a1aafd997535569b P 7e494085174f441a89e020fdd2653195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f97366d0c39743b3b0ae64e54d652a45" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 46199 } } peers { permanent_uuid: "4a6080c03f914d8d9a8210c01a6dbab6" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37139 } } peers { permanent_uuid: "7e494085174f441a89e020fdd2653195" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46777 } }
I20260927 10:53:33.387619 19168 raft_consensus.cc:374] T 9b02cbfd9e734c30a1aafd997535569b P 7e494085174f441a89e020fdd2653195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:33.387630 19170 tablet_bootstrap.cc:492] T 9b02cbfd9e734c30a1aafd997535569b P 4a6080c03f914d8d9a8210c01a6dbab6: No bootstrap required, opened a new log
I20260927 10:53:33.387718 19168 raft_consensus.cc:729] T 9b02cbfd9e734c30a1aafd997535569b P 7e494085174f441a89e020fdd2653195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e494085174f441a89e020fdd2653195, State: Initialized, Role: FOLLOWER
I20260927 10:53:33.387827 19170 ts_tablet_manager.cc:1397] T 9b02cbfd9e734c30a1aafd997535569b P 4a6080c03f914d8d9a8210c01a6dbab6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:53:33.388213 19168 consensus_queue.cc:260] T 9b02cbfd9e734c30a1aafd997535569b P 7e494085174f441a89e020fdd2653195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f97366d0c39743b3b0ae64e54d652a45" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 46199 } } peers { permanent_uuid: "4a6080c03f914d8d9a8210c01a6dbab6" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37139 } } peers { permanent_uuid: "7e494085174f441a89e020fdd2653195" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46777 } }
I20260927 10:53:33.388849 19151 heartbeater.cc:503] Master 127.14.68.62:41189 was elected leader, sending a full tablet report...
I20260927 10:53:33.389204 19168 ts_tablet_manager.cc:1428] T 9b02cbfd9e734c30a1aafd997535569b P 7e494085174f441a89e020fdd2653195: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:53:33.389225 19169 raft_consensus.cc:348] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f97366d0c39743b3b0ae64e54d652a45" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 46199 } } peers { permanent_uuid: "4a6080c03f914d8d9a8210c01a6dbab6" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37139 } } peers { permanent_uuid: "7e494085174f441a89e020fdd2653195" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46777 } }
I20260927 10:53:33.389495 19169 raft_consensus.cc:374] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:33.389575 19169 raft_consensus.cc:729] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f97366d0c39743b3b0ae64e54d652a45, State: Initialized, Role: FOLLOWER
I20260927 10:53:33.390048 19169 consensus_queue.cc:260] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f97366d0c39743b3b0ae64e54d652a45" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 46199 } } peers { permanent_uuid: "4a6080c03f914d8d9a8210c01a6dbab6" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37139 } } peers { permanent_uuid: "7e494085174f441a89e020fdd2653195" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46777 } }
I20260927 10:53:33.390823 18885 heartbeater.cc:503] Master 127.14.68.62:41189 was elected leader, sending a full tablet report...
I20260927 10:53:33.391011 19169 ts_tablet_manager.cc:1428] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:53:33.394158 19170 raft_consensus.cc:348] T 9b02cbfd9e734c30a1aafd997535569b P 4a6080c03f914d8d9a8210c01a6dbab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f97366d0c39743b3b0ae64e54d652a45" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 46199 } } peers { permanent_uuid: "4a6080c03f914d8d9a8210c01a6dbab6" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37139 } } peers { permanent_uuid: "7e494085174f441a89e020fdd2653195" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46777 } }
I20260927 10:53:33.394366 19170 raft_consensus.cc:374] T 9b02cbfd9e734c30a1aafd997535569b P 4a6080c03f914d8d9a8210c01a6dbab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:33.394428 19170 raft_consensus.cc:729] T 9b02cbfd9e734c30a1aafd997535569b P 4a6080c03f914d8d9a8210c01a6dbab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6080c03f914d8d9a8210c01a6dbab6, State: Initialized, Role: FOLLOWER
I20260927 10:53:33.394764 19170 consensus_queue.cc:260] T 9b02cbfd9e734c30a1aafd997535569b P 4a6080c03f914d8d9a8210c01a6dbab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f97366d0c39743b3b0ae64e54d652a45" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 46199 } } peers { permanent_uuid: "4a6080c03f914d8d9a8210c01a6dbab6" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37139 } } peers { permanent_uuid: "7e494085174f441a89e020fdd2653195" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46777 } }
I20260927 10:53:33.395303 19018 heartbeater.cc:503] Master 127.14.68.62:41189 was elected leader, sending a full tablet report...
I20260927 10:53:33.395525 19170 ts_tablet_manager.cc:1428] T 9b02cbfd9e734c30a1aafd997535569b P 4a6080c03f914d8d9a8210c01a6dbab6: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
W20260927 10:53:33.559756 19152 tablet.cc:2369] T 9b02cbfd9e734c30a1aafd997535569b P 7e494085174f441a89e020fdd2653195: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:53:33.597700 19019 tablet.cc:2369] T 9b02cbfd9e734c30a1aafd997535569b P 4a6080c03f914d8d9a8210c01a6dbab6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:53:33.608026 18886 tablet.cc:2369] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:53:33.737406 19175 raft_consensus.cc:482] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:33.737727 19175 raft_consensus.cc:504] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f97366d0c39743b3b0ae64e54d652a45" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 46199 } } peers { permanent_uuid: "4a6080c03f914d8d9a8210c01a6dbab6" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37139 } } peers { permanent_uuid: "7e494085174f441a89e020fdd2653195" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46777 } }
I20260927 10:53:33.738996 19175 leader_election.cc:288] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a6080c03f914d8d9a8210c01a6dbab6 (127.14.68.2:37139), 7e494085174f441a89e020fdd2653195 (127.14.68.3:46777)
I20260927 10:53:33.743248 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:33.739401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:35349 (local address 127.14.68.3:46777)
0927 10:53:33.739519 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:33.739524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:33.739817 (+ 293us) server_negotiation.cc:407] Connection header received
0927 10:53:33.740048 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:33.740051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:33.740122 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:33.740204 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:33.740550 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.741110 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.742269 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.742667 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.742748 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:33.742838 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:33.743033 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 10:53:33.743121 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:53:33.743275 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:33.739290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:46777 (local address 127.14.68.1:35349)
0927 10:53:33.739665 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:33.739678 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:33.739885 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:33.740242 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:33.740245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:33.740260 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:33.740378 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:33.740404 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.741216 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.741221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:33.742137 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:33.742146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.742796 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.742800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:33.742900 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:33.742927 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:53:33.743001 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 10:53:33.743054 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":22,"spinlock_wait_cycles":13568,"thread_start_us":125,"threads_started":1}
I20260927 10:53:33.743325 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:33.739384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:46333 (local address 127.14.68.2:37139)
0927 10:53:33.739519 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:33.739524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:33.739811 (+ 287us) server_negotiation.cc:407] Connection header received
0927 10:53:33.740007 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:33.740010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:33.740067 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:33.740126 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:33.740443 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.741197 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.742301 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.742695 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.742823 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:33.742885 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:33.743125 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 10:53:33.743200 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:53:33.743295 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:33.739290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:37139 (local address 127.14.68.1:46333)
0927 10:53:33.739676 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:33.739688 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:33.739886 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:33.740168 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:33.740173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:33.740195 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:33.740335 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:33.740341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.741316 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.741319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:33.742144 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:33.742153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.742851 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.742858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:33.742957 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:33.743000 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 10:53:33.743087 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 10:53:33.743149 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":4,"spinlock_wait_cycles":33792,"thread_start_us":108,"threads_started":1}
I20260927 10:53:33.744092 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b02cbfd9e734c30a1aafd997535569b" candidate_uuid: "f97366d0c39743b3b0ae64e54d652a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6080c03f914d8d9a8210c01a6dbab6" is_pre_election: true
I20260927 10:53:33.744161 19105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b02cbfd9e734c30a1aafd997535569b" candidate_uuid: "f97366d0c39743b3b0ae64e54d652a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e494085174f441a89e020fdd2653195" is_pre_election: true
I20260927 10:53:33.744405 18972 raft_consensus.cc:2454] T 9b02cbfd9e734c30a1aafd997535569b P 4a6080c03f914d8d9a8210c01a6dbab6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f97366d0c39743b3b0ae64e54d652a45 in term 0.
I20260927 10:53:33.744442 19105 raft_consensus.cc:2454] T 9b02cbfd9e734c30a1aafd997535569b P 7e494085174f441a89e020fdd2653195 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f97366d0c39743b3b0ae64e54d652a45 in term 0.
I20260927 10:53:33.744892 18773 leader_election.cc:302] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6080c03f914d8d9a8210c01a6dbab6, f97366d0c39743b3b0ae64e54d652a45; no voters:
I20260927 10:53:33.745184 19175 raft_consensus.cc:2781] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:33.745265 19175 raft_consensus.cc:482] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:33.745322 19175 raft_consensus.cc:3037] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:33.746377 19175 raft_consensus.cc:504] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f97366d0c39743b3b0ae64e54d652a45" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 46199 } } peers { permanent_uuid: "4a6080c03f914d8d9a8210c01a6dbab6" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37139 } } peers { permanent_uuid: "7e494085174f441a89e020fdd2653195" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46777 } }
I20260927 10:53:33.746752 19175 leader_election.cc:288] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45 [CANDIDATE]: Term 1 election: Requested vote from peers 4a6080c03f914d8d9a8210c01a6dbab6 (127.14.68.2:37139), 7e494085174f441a89e020fdd2653195 (127.14.68.3:46777)
I20260927 10:53:33.747113 19105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b02cbfd9e734c30a1aafd997535569b" candidate_uuid: "f97366d0c39743b3b0ae64e54d652a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e494085174f441a89e020fdd2653195"
I20260927 10:53:33.747128 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b02cbfd9e734c30a1aafd997535569b" candidate_uuid: "f97366d0c39743b3b0ae64e54d652a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6080c03f914d8d9a8210c01a6dbab6"
I20260927 10:53:33.747253 19105 raft_consensus.cc:3037] T 9b02cbfd9e734c30a1aafd997535569b P 7e494085174f441a89e020fdd2653195 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:33.747292 18972 raft_consensus.cc:3037] T 9b02cbfd9e734c30a1aafd997535569b P 4a6080c03f914d8d9a8210c01a6dbab6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:33.748368 19105 raft_consensus.cc:2454] T 9b02cbfd9e734c30a1aafd997535569b P 7e494085174f441a89e020fdd2653195 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f97366d0c39743b3b0ae64e54d652a45 in term 1.
I20260927 10:53:33.748374 18972 raft_consensus.cc:2454] T 9b02cbfd9e734c30a1aafd997535569b P 4a6080c03f914d8d9a8210c01a6dbab6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f97366d0c39743b3b0ae64e54d652a45 in term 1.
I20260927 10:53:33.748708 18775 leader_election.cc:302] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e494085174f441a89e020fdd2653195, f97366d0c39743b3b0ae64e54d652a45; no voters:
I20260927 10:53:33.748960 19175 raft_consensus.cc:2781] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:33.749171 19175 raft_consensus.cc:686] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45 [term 1 LEADER]: Becoming Leader. State: Replica: f97366d0c39743b3b0ae64e54d652a45, State: Running, Role: LEADER
I20260927 10:53:33.749522 19175 consensus_queue.cc:237] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f97366d0c39743b3b0ae64e54d652a45" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 46199 } } peers { permanent_uuid: "4a6080c03f914d8d9a8210c01a6dbab6" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37139 } } peers { permanent_uuid: "7e494085174f441a89e020fdd2653195" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46777 } }
I20260927 10:53:33.752673 18694 catalog_manager.cc:5495] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45 reported cstate change: term changed from 0 to 1, leader changed from <none> to f97366d0c39743b3b0ae64e54d652a45 (127.14.68.1). New cstate: current_term: 1 leader_uuid: "f97366d0c39743b3b0ae64e54d652a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f97366d0c39743b3b0ae64e54d652a45" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 46199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a6080c03f914d8d9a8210c01a6dbab6" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e494085174f441a89e020fdd2653195" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46777 } health_report { overall_health: UNKNOWN } } }
I20260927 10:53:33.816509 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:33.812855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50444 (local address 127.14.68.1:46199)
0927 10:53:33.812998 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:33.813002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:33.813017 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:53:33.813143 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:33.813145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:33.813200 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:33.813319 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:33.813566 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.814070 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.814935 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:33.815153 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:33.815286 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:33.815616 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:33.816115 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:33.816263 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:33.816295 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 10:53:33.816357 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:53:33.822642 19105 raft_consensus.cc:1261] T 9b02cbfd9e734c30a1aafd997535569b P 7e494085174f441a89e020fdd2653195 [term 1 FOLLOWER]: Refusing update from remote peer f97366d0c39743b3b0ae64e54d652a45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:33.822642 18972 raft_consensus.cc:1261] T 9b02cbfd9e734c30a1aafd997535569b P 4a6080c03f914d8d9a8210c01a6dbab6 [term 1 FOLLOWER]: Refusing update from remote peer f97366d0c39743b3b0ae64e54d652a45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:33.823414 19179 consensus_queue.cc:1035] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e494085174f441a89e020fdd2653195" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 46777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:33.823670 19175 consensus_queue.cc:1035] T 9b02cbfd9e734c30a1aafd997535569b P f97366d0c39743b3b0ae64e54d652a45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6080c03f914d8d9a8210c01a6dbab6" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:33.831075 19183 mvcc.cc:205] Tried to move back new op lower bound from 7333914271010263040 to 7333914270720712704. Current Snapshot: MvccSnapshot[applied={T|T < 7333914271010263040}]
I20260927 10:53:33.831265 19184 mvcc.cc:205] Tried to move back new op lower bound from 7333914271010263040 to 7333914270720712704. Current Snapshot: MvccSnapshot[applied={T|T < 7333914271010263040}]
I20260927 10:53:33.831465 19182 mvcc.cc:205] Tried to move back new op lower bound from 7333914271010263040 to 7333914270720712704. Current Snapshot: MvccSnapshot[applied={T|T < 7333914271010263040}]
I20260927 10:53:33.839473 18695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:33.839663 18695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:33.842186 18695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ad75c1148e144fcbbef53b8b81b3dac: Sending DeleteTablet for 3 replicas of tablet 9b02cbfd9e734c30a1aafd997535569b
I20260927 10:53:33.842898 18952 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b02cbfd9e734c30a1aafd997535569b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:33 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:46760
I20260927 10:53:33.843154 19085 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b02cbfd9e734c30a1aafd997535569b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:33 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:34288
I20260927 10:53:33.843406 19192 tablet_replica.cc:330] T 9b02cbfd9e734c30a1aafd997535569b P 4a6080c03f914d8d9a8210c01a6dbab6: stopping tablet replica
I20260927 10:53:33.843540 18819 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b02cbfd9e734c30a1aafd997535569b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:33 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:50438
I20260927 10:53:33.843640 19193 tablet_replica.cc:330] T 9b02cbfd9e734c30a1aafd997535569b P 7e494085174f441a89e020fdd2653195: stopping tablet replica
I20260927 10:53:33.843793 19192 raft_consensus.cc:2229] T 9b02cbfd9e734c30a1aafd997535569b P 4a6080c03f914d8d9a8210c01a6dbab6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:33.843957 19193 raft_consensus.cc:2229] T 9b02cbfd9e734c30a1aafd997535569b P 7e494085174f441a89e020fdd2653195 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:33.844106 19192 raft_consensus.cc:2258] T 9b02cbfd9e734c30a1aafd997535569b P 4a6080c03f914d8d9a8210c01a6dbab6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:33.844307 19193 raft_consensus.cc:2258] T 9b02cbfd9e734c30a1aafd997535569b P 7e494085174f441a89e020fdd2653195 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:33.844460 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 18757
I20260927 10:53:33.846221 19193 ts_tablet_manager.cc:1905] T 9b02cbfd9e734c30a1aafd997535569b P 7e494085174f441a89e020fdd2653195: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:33.846967 19192 ts_tablet_manager.cc:1905] T 9b02cbfd9e734c30a1aafd997535569b P 4a6080c03f914d8d9a8210c01a6dbab6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:33.849339 19192 ts_tablet_manager.cc:1918] T 9b02cbfd9e734c30a1aafd997535569b P 4a6080c03f914d8d9a8210c01a6dbab6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:33.849457 19192 log.cc:1196] T 9b02cbfd9e734c30a1aafd997535569b P 4a6080c03f914d8d9a8210c01a6dbab6: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/9b02cbfd9e734c30a1aafd997535569b
I20260927 10:53:33.849838 19192 ts_tablet_manager.cc:1939] T 9b02cbfd9e734c30a1aafd997535569b P 4a6080c03f914d8d9a8210c01a6dbab6: Deleting consensus metadata
I20260927 10:53:33.849900 19193 ts_tablet_manager.cc:1918] T 9b02cbfd9e734c30a1aafd997535569b P 7e494085174f441a89e020fdd2653195: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:33.850023 19193 log.cc:1196] T 9b02cbfd9e734c30a1aafd997535569b P 7e494085174f441a89e020fdd2653195: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/9b02cbfd9e734c30a1aafd997535569b
I20260927 10:53:33.850335 19193 ts_tablet_manager.cc:1939] T 9b02cbfd9e734c30a1aafd997535569b P 7e494085174f441a89e020fdd2653195: Deleting consensus metadata
I20260927 10:53:33.851184 18680 catalog_manager.cc:4841] TS 4a6080c03f914d8d9a8210c01a6dbab6 (127.14.68.2:37139): tablet 9b02cbfd9e734c30a1aafd997535569b (table test-table [id=23bf90c7b93a4108a0c88b03fb686775]) successfully deleted
I20260927 10:53:33.851496 18682 catalog_manager.cc:4841] TS 7e494085174f441a89e020fdd2653195 (127.14.68.3:46777): tablet 9b02cbfd9e734c30a1aafd997535569b (table test-table [id=23bf90c7b93a4108a0c88b03fb686775]) successfully deleted
W20260927 10:53:33.852077 18681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.68.1:46199 (error 108)
I20260927 10:53:33.852264 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 18890
I20260927 10:53:33.853976 19165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:33.853251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:46199 (local address 127.0.0.1:50448)
0927 10:53:33.853766 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:33.853850 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.1:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288}
W20260927 10:53:33.854146 18681 catalog_manager.cc:4584] TS f97366d0c39743b3b0ae64e54d652a45 (127.14.68.1:46199): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9b02cbfd9e734c30a1aafd997535569b: Network error: Client connection negotiation failed: client connection to 127.14.68.1:46199: connect: Connection refused (error 111)
I20260927 10:53:33.858503 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 19023
I20260927 10:53:33.864034 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 18664
2026-09-27T10:53:33Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2617 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T10:53:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:33Z Disabled control of system clock
I20260927 10:53:33.893401 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:33495
--webserver_interface=127.14.68.62
--webserver_port=0
--builtin_ntp_servers=127.14.68.20:33911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:33495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 10:53:34.002002 19199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:34.002269 19199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:34.002321 19199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:34.006251 19199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:34.006300 19199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:34.006315 19199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:34.006331 19199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:34.006345 19199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:34.010927 19199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:33911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:33495
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:33495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:34.011874 19199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:34.013145 19199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:34.021080 19207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:34.021080 19205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:34.021186 19204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:34.021590 19199 server_base.cc:1034] running on GCE node
I20260927 10:53:34.022109 19199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:34.023157 19199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:34.024351 19199 hybrid_clock.cc:648] HybridClock initialized: now 1790506414024326 us; error 42 us; skew 500 ppm
I20260927 10:53:34.026152 19199 webserver.cc:460] Webserver started at http://127.14.68.62:39043/ using document root <none> and password file <none>
I20260927 10:53:34.026772 19199 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:34.026870 19199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:34.027102 19199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:34.029301 19199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "81b8baffe4a24759b6c00f28ad6b3a86"
format_stamp: "Formatted at 2026-09-27 10:53:34 on dist-test-slave-tz51"
server_key: "124b245297c364a30fb7658e2e7557f4"
server_key_iv: "3e3f2ce12e365d49099a091cce991e13"
server_key_version: "encryptionkey@0"
I20260927 10:53:34.029874 19199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "81b8baffe4a24759b6c00f28ad6b3a86"
format_stamp: "Formatted at 2026-09-27 10:53:34 on dist-test-slave-tz51"
server_key: "124b245297c364a30fb7658e2e7557f4"
server_key_iv: "3e3f2ce12e365d49099a091cce991e13"
server_key_version: "encryptionkey@0"
I20260927 10:53:34.033555 19199 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:53:34.036032 19213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:34.037256 19199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:53:34.037397 19199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "81b8baffe4a24759b6c00f28ad6b3a86"
format_stamp: "Formatted at 2026-09-27 10:53:34 on dist-test-slave-tz51"
server_key: "124b245297c364a30fb7658e2e7557f4"
server_key_iv: "3e3f2ce12e365d49099a091cce991e13"
server_key_version: "encryptionkey@0"
I20260927 10:53:34.037519 19199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:34.042243 19199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:34.042831 19199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:34.043006 19199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:34.049964 19199 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:33495
I20260927 10:53:34.049993 19265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:33495 every 8 connection(s)
I20260927 10:53:34.050820 19199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:34.053867 19266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:34.058544 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 19199
I20260927 10:53:34.058679 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:53:34.058921 14608 external_mini_cluster.cc:1428] Setting key 38610e78bde94e89259d4fa4045f7dde
I20260927 10:53:34.059129 19266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86: Bootstrap starting.
I20260927 10:53:34.061600 19266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:34.062284 19266 log.cc:824] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:34.064260 19269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.060201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42322 (local address 127.14.68.62:33495)
0927 10:53:34.060822 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:34.060836 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:34.060871 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:53:34.061451 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:34.061467 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:34.061863 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:34.062180 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:34.062893 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:34.062901 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:34.062929 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:34.062941 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:34.063018 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:34.063043 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:34.063159 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:34.063297 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 10:53:34.063554 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":132,"threads_started":1}
I20260927 10:53:34.064527 19266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86: No bootstrap required, opened a new log
I20260927 10:53:34.068137 19266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b8baffe4a24759b6c00f28ad6b3a86" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33495 } }
I20260927 10:53:34.068369 19266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:34.068449 19266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b8baffe4a24759b6c00f28ad6b3a86, State: Initialized, Role: FOLLOWER
I20260927 10:53:34.068879 19266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b8baffe4a24759b6c00f28ad6b3a86" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33495 } }
I20260927 10:53:34.069022 19266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:34.069106 19266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:34.069201 19266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:34.070005 19266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b8baffe4a24759b6c00f28ad6b3a86" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33495 } }
I20260927 10:53:34.070319 19266 leader_election.cc:302] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81b8baffe4a24759b6c00f28ad6b3a86; no voters:
I20260927 10:53:34.070638 19266 leader_election.cc:288] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:34.070698 19271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:34.070904 19271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86 [term 1 LEADER]: Becoming Leader. State: Replica: 81b8baffe4a24759b6c00f28ad6b3a86, State: Running, Role: LEADER
I20260927 10:53:34.071169 19271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b8baffe4a24759b6c00f28ad6b3a86" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33495 } }
I20260927 10:53:34.071844 19266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:34.072806 19273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81b8baffe4a24759b6c00f28ad6b3a86. Latest consensus state: current_term: 1 leader_uuid: "81b8baffe4a24759b6c00f28ad6b3a86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b8baffe4a24759b6c00f28ad6b3a86" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33495 } } }
I20260927 10:53:34.072847 19272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81b8baffe4a24759b6c00f28ad6b3a86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b8baffe4a24759b6c00f28ad6b3a86" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33495 } } }
I20260927 10:53:34.072927 19273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:34.072963 19272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:34.073279 19280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:34.076279 19280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:34.081526 19280 catalog_manager.cc:1347] Generated new cluster ID: 4434a46e5c014cc7a201bf15b1987d76
I20260927 10:53:34.081584 19280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:34.130015 19280 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:34.130371 19280 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:34.143469 19280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86: Generated new TSK 0
I20260927 10:53:34.144359 19280 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:53:34.158874 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:33495
--builtin_ntp_servers=127.14.68.20:33911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 10:53:34.266723 19290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:34.266976 19290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:34.267015 19290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:34.271008 19290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:34.271067 19290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:34.271121 19290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:53:34.276105 19290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:33911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:33495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:34.277187 19290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:34.278587 19290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:34.288100 19296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:34.288116 19290 server_base.cc:1034] running on GCE node
W20260927 10:53:34.288137 19298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:34.288134 19295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:34.288661 19290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:34.289255 19290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:34.290438 19290 hybrid_clock.cc:648] HybridClock initialized: now 1790506414290432 us; error 32 us; skew 500 ppm
I20260927 10:53:34.292236 19290 webserver.cc:460] Webserver started at http://127.14.68.1:37737/ using document root <none> and password file <none>
I20260927 10:53:34.292780 19290 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:34.292872 19290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:34.293095 19290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:34.295060 19290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "6dcafbebe4f64bb184e76ccdac26fd1b"
format_stamp: "Formatted at 2026-09-27 10:53:34 on dist-test-slave-tz51"
server_key: "2c30215260db269792ad46b2c3b9f95c"
server_key_iv: "75925aa28f6b4ba87c774c540f13a970"
server_key_version: "encryptionkey@0"
I20260927 10:53:34.295621 19290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "6dcafbebe4f64bb184e76ccdac26fd1b"
format_stamp: "Formatted at 2026-09-27 10:53:34 on dist-test-slave-tz51"
server_key: "2c30215260db269792ad46b2c3b9f95c"
server_key_iv: "75925aa28f6b4ba87c774c540f13a970"
server_key_version: "encryptionkey@0"
I20260927 10:53:34.299037 19290 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:53:34.301524 19304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:34.302748 19290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:34.302884 19290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "6dcafbebe4f64bb184e76ccdac26fd1b"
format_stamp: "Formatted at 2026-09-27 10:53:34 on dist-test-slave-tz51"
server_key: "2c30215260db269792ad46b2c3b9f95c"
server_key_iv: "75925aa28f6b4ba87c774c540f13a970"
server_key_version: "encryptionkey@0"
I20260927 10:53:34.303006 19290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:34.307397 19290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:34.308022 19290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:34.308204 19290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:34.308749 19290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:34.309649 19290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:34.309720 19290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:34.309787 19290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:34.309836 19290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:34.319119 19290 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:34995
I20260927 10:53:34.319150 19417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:34995 every 8 connection(s)
I20260927 10:53:34.320113 19290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:53:34.324894 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 19290
I20260927 10:53:34.325018 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:53:34.325374 14608 external_mini_cluster.cc:1428] Setting key 061a0b784af10cbdb8876c98e993d376
I20260927 10:53:34.326442 19269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.321992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:58379 (local address 127.14.68.62:33495)
0927 10:53:34.322142 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:34.322147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:34.323249 (+ 1102us) server_negotiation.cc:407] Connection header received
0927 10:53:34.324564 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:34.324566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:34.324603 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:34.324667 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:34.325973 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:34.325977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:34.325980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:34.325992 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:34.326014 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:34.326024 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:34.326136 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:34.326302 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 10:53:34.326338 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:53:34.327316 19420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.322445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:33495 (local address 127.14.68.1:58379)
0927 10:53:34.323080 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:34.323125 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:34.324378 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:34.324878 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:34.324888 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:34.325013 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:34.325649 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:53:34.325651 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:34.325691 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:53:34.325797 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:34.326138 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:34.326147 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:34.326152 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 10:53:34.326339 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 10:53:34.326572 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":98,"threads_started":1}
I20260927 10:53:34.328227 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:33495
--builtin_ntp_servers=127.14.68.20:33911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 10:53:34.328823 19418 heartbeater.cc:347] Connected to a master server at 127.14.68.62:33495
I20260927 10:53:34.329063 19418 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:34.329540 19418 heartbeater.cc:511] Master 127.14.68.62:33495 requested a full tablet report, sending...
I20260927 10:53:34.331183 19230 ts_manager.cc:194] Registered new tserver with Master: 6dcafbebe4f64bb184e76ccdac26fd1b (127.14.68.1:34995)
W20260927 10:53:34.438359 19421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:34.438632 19421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:34.438702 19421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:34.442960 19421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:34.443029 19421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:34.443117 19421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:53:34.447893 19421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:33911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:33495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:34.449045 19421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:34.450459 19421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:34.460083 19426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:34.460120 19429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:34.460098 19427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:34.460245 19421 server_base.cc:1034] running on GCE node
I20260927 10:53:34.460935 19421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:34.461539 19421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:34.462700 19421 hybrid_clock.cc:648] HybridClock initialized: now 1790506414462684 us; error 28 us; skew 500 ppm
I20260927 10:53:34.464591 19421 webserver.cc:460] Webserver started at http://127.14.68.2:37035/ using document root <none> and password file <none>
I20260927 10:53:34.465155 19421 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:34.465217 19421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:34.465432 19421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:34.467403 19421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "d4c2254259c14c72a909512e134c94ec"
format_stamp: "Formatted at 2026-09-27 10:53:34 on dist-test-slave-tz51"
server_key: "1aa871109c9a787b219df6c8938b1e1e"
server_key_iv: "c98fd1d3935abb748799f2536bd91bb4"
server_key_version: "encryptionkey@0"
I20260927 10:53:34.467954 19421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "d4c2254259c14c72a909512e134c94ec"
format_stamp: "Formatted at 2026-09-27 10:53:34 on dist-test-slave-tz51"
server_key: "1aa871109c9a787b219df6c8938b1e1e"
server_key_iv: "c98fd1d3935abb748799f2536bd91bb4"
server_key_version: "encryptionkey@0"
I20260927 10:53:34.471422 19421 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:53:34.474006 19435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:34.475243 19421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:53:34.475355 19421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "d4c2254259c14c72a909512e134c94ec"
format_stamp: "Formatted at 2026-09-27 10:53:34 on dist-test-slave-tz51"
server_key: "1aa871109c9a787b219df6c8938b1e1e"
server_key_iv: "c98fd1d3935abb748799f2536bd91bb4"
server_key_version: "encryptionkey@0"
I20260927 10:53:34.475455 19421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:34.479807 19421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:34.480425 19421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:34.480605 19421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:34.481117 19421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:34.481987 19421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:34.482033 19421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:34.482095 19421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:34.482129 19421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:34.490761 19421 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:37713
I20260927 10:53:34.490803 19548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:37713 every 8 connection(s)
I20260927 10:53:34.491617 19421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:53:34.494879 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 19421
I20260927 10:53:34.494979 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:53:34.495283 14608 external_mini_cluster.cc:1428] Setting key 30825b3ab6b052510bb7dce2b9a13434
I20260927 10:53:34.497589 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:33495
--builtin_ntp_servers=127.14.68.20:33911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 10:53:34.497780 19269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.493394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:60073 (local address 127.14.68.62:33495)
0927 10:53:34.493549 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:34.493553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:34.494493 (+ 940us) server_negotiation.cc:407] Connection header received
0927 10:53:34.495979 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:34.495982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:34.496018 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:34.496117 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:34.497273 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:34.497276 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:34.497279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:34.497289 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:34.497309 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:34.497317 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:34.497387 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:34.497563 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 10:53:34.497601 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 10:53:34.498848 19551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.493732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:33495 (local address 127.14.68.2:60073)
0927 10:53:34.494363 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:34.494404 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:34.495687 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:34.496260 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:34.496268 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:34.496356 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:34.496968 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:53:34.496969 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:34.497020 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:53:34.497122 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:34.497406 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:34.497417 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:34.497424 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 10:53:34.497603 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 10:53:34.497859 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":102,"threads_started":1}
I20260927 10:53:34.500095 19549 heartbeater.cc:347] Connected to a master server at 127.14.68.62:33495
I20260927 10:53:34.500330 19549 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:34.500715 19549 heartbeater.cc:511] Master 127.14.68.62:33495 requested a full tablet report, sending...
I20260927 10:53:34.501749 19230 ts_manager.cc:194] Registered new tserver with Master: d4c2254259c14c72a909512e134c94ec (127.14.68.2:37713)
W20260927 10:53:34.610577 19552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:34.610879 19552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:34.610953 19552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:34.615046 19552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:34.615136 19552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:34.615227 19552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:53:34.619988 19552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:33911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:33495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:34.621101 19552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:34.622565 19552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:34.632031 19558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:34.632172 19552 server_base.cc:1034] running on GCE node
W20260927 10:53:34.632047 19560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:34.632053 19557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:34.632789 19552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:34.633419 19552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:34.634578 19552 hybrid_clock.cc:648] HybridClock initialized: now 1790506414634536 us; error 58 us; skew 500 ppm
I20260927 10:53:34.636536 19552 webserver.cc:460] Webserver started at http://127.14.68.3:32981/ using document root <none> and password file <none>
I20260927 10:53:34.637112 19552 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:34.637180 19552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:34.637434 19552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:34.639477 19552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "a578b8d774bf449f83cbb84f7979918a"
format_stamp: "Formatted at 2026-09-27 10:53:34 on dist-test-slave-tz51"
server_key: "cbd9f91d06f6882377d76e8b11555e89"
server_key_iv: "14d857e9f69c7d1a2102d0ec641fac2b"
server_key_version: "encryptionkey@0"
I20260927 10:53:34.640095 19552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "a578b8d774bf449f83cbb84f7979918a"
format_stamp: "Formatted at 2026-09-27 10:53:34 on dist-test-slave-tz51"
server_key: "cbd9f91d06f6882377d76e8b11555e89"
server_key_iv: "14d857e9f69c7d1a2102d0ec641fac2b"
server_key_version: "encryptionkey@0"
I20260927 10:53:34.644127 19552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 10:53:34.646636 19566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:34.647904 19552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:34.648018 19552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "a578b8d774bf449f83cbb84f7979918a"
format_stamp: "Formatted at 2026-09-27 10:53:34 on dist-test-slave-tz51"
server_key: "cbd9f91d06f6882377d76e8b11555e89"
server_key_iv: "14d857e9f69c7d1a2102d0ec641fac2b"
server_key_version: "encryptionkey@0"
I20260927 10:53:34.648162 19552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:34.652642 19552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:34.653223 19552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:34.653396 19552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:34.653951 19552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:34.654841 19552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:34.654887 19552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:34.654950 19552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:34.654989 19552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:34.663478 19552 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:44767
I20260927 10:53:34.663501 19679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:44767 every 8 connection(s)
I20260927 10:53:34.664456 19552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:53:34.670331 19269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.666244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:46233 (local address 127.14.68.62:33495)
0927 10:53:34.666381 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:34.666385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:34.667364 (+ 979us) server_negotiation.cc:407] Connection header received
0927 10:53:34.668652 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:34.668655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:34.668696 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:34.668772 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:34.669911 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:34.669913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:34.669915 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:34.669925 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:34.669941 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:34.669949 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:34.670071 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:34.670180 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 10:53:34.670220 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:53:34.671238 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.666580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:33495 (local address 127.14.68.3:46233)
0927 10:53:34.667218 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:34.667258 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:34.668449 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:34.668935 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:34.668942 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:34.669034 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:34.669658 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:53:34.669659 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:34.669701 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:53:34.669779 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:34.670057 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:34.670063 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:34.670067 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:53:34.670199 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 10:53:34.670451 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":103,"threads_started":1}
I20260927 10:53:34.672500 19680 heartbeater.cc:347] Connected to a master server at 127.14.68.62:33495
I20260927 10:53:34.672766 19680 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:34.673210 19680 heartbeater.cc:511] Master 127.14.68.62:33495 requested a full tablet report, sending...
I20260927 10:53:34.674228 19230 ts_manager.cc:194] Registered new tserver with Master: a578b8d774bf449f83cbb84f7979918a (127.14.68.3:44767)
I20260927 10:53:34.674438 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 19552
I20260927 10:53:34.674571 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:53:34.674857 14608 external_mini_cluster.cc:1428] Setting key e1f3d3372cdca2095dfd44a13b7f74a3
I20260927 10:53:34.676756 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:53:34.680284 19269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.678635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42328 (local address 127.14.68.62:33495)
0927 10:53:34.678755 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:34.678759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:34.678984 (+ 225us) server_negotiation.cc:407] Connection header received
0927 10:53:34.679105 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:34.679108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:34.679151 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:34.679241 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:34.679896 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:34.679900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:34.679903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:34.679914 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:34.679933 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:34.679942 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:34.680038 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:34.680108 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 10:53:34.680154 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:53:34.682739 19230 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:53:34.685711 19230 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:34.696796 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.693541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:34995 (local address 127.0.0.1:51590)
0927 10:53:34.694053 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:34.694098 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:34.694209 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:34.695325 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:34.695328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:34.695346 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:34.695949 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:53:34.695953 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:34.695973 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:53:34.696035 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:34.696457 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:34.696466 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:34.696472 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:53:34.696584 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 10:53:34.696660 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":99,"threads_started":1}
I20260927 10:53:34.696825 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.693692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51590 (local address 127.14.68.1:34995)
0927 10:53:34.694880 (+ 1188us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:34.694892 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:34.694914 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:53:34.694998 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:34.695004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:34.695184 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:34.695553 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:34.696262 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:34.696270 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:34.696285 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:34.696298 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:34.696338 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:34.696366 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:34.696477 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:34.696623 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 10:53:34.696694 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1078,"thread_start_us":116,"threads_started":1}
I20260927 10:53:34.697134 19692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.694251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:44767 (local address 127.0.0.1:46694)
0927 10:53:34.694929 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:34.694943 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:34.695025 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:34.695978 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:34.695981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:34.695995 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:34.696434 (+ 439us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:53:34.696436 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:34.696446 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:53:34.696469 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:34.696838 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:34.696840 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:34.696843 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 10:53:34.696937 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 10:53:34.696977 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":94,"threads_started":1}
I20260927 10:53:34.697507 19694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.694213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58474 (local address 127.14.68.2:37713)
0927 10:53:34.694622 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:34.694635 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:34.694661 (+ 26us) server_negotiation.cc:407] Connection header received
0927 10:53:34.694792 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:34.694799 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:34.695031 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:34.695175 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:34.696486 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:34.696494 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:34.696506 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:34.696520 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:34.696567 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:34.696596 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:34.696698 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:34.697295 (+ 597us) server_negotiation.cc:299] Negotiation successful
0927 10:53:34.697369 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":112,"threads_started":1}
I20260927 10:53:34.697528 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.694500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46694 (local address 127.14.68.3:44767)
0927 10:53:34.694928 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:34.694938 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:34.695035 (+ 97us) server_negotiation.cc:407] Connection header received
0927 10:53:34.695100 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:34.695107 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:34.695279 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:34.695396 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:34.696596 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:34.696601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:34.696610 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:34.696624 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:34.696664 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:34.696690 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:34.696797 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:34.697341 (+ 544us) server_negotiation.cc:299] Negotiation successful
0927 10:53:34.697395 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":158,"threads_started":1}
I20260927 10:53:34.697901 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.693725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:37713 (local address 127.0.0.1:58474)
0927 10:53:34.694549 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:34.694562 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:34.694653 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:34.695161 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:34.695167 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:34.695263 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:34.696042 (+ 779us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:53:34.696045 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:34.696056 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:53:34.696085 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:34.696743 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:34.696748 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:34.696754 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:53:34.697113 (+ 359us) client_negotiation.cc:240] Negotiation successful
0927 10:53:34.697144 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":97,"threads_started":1}
I20260927 10:53:34.699755 19352 tablet_service.cc:1467] Processing CreateTablet for tablet e27ae90b301745d5b6427ade46442dd8 (DEFAULT_TABLE table=test-table [id=40bbd22bebf04bcc89b2602815ae811e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:34.699946 19614 tablet_service.cc:1467] Processing CreateTablet for tablet e27ae90b301745d5b6427ade46442dd8 (DEFAULT_TABLE table=test-table [id=40bbd22bebf04bcc89b2602815ae811e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:34.700820 19614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e27ae90b301745d5b6427ade46442dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:34.700840 19352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e27ae90b301745d5b6427ade46442dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:34.700650 19483 tablet_service.cc:1467] Processing CreateTablet for tablet e27ae90b301745d5b6427ade46442dd8 (DEFAULT_TABLE table=test-table [id=40bbd22bebf04bcc89b2602815ae811e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:34.701483 19483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e27ae90b301745d5b6427ade46442dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:34.707497 19697 tablet_bootstrap.cc:492] T e27ae90b301745d5b6427ade46442dd8 P a578b8d774bf449f83cbb84f7979918a: Bootstrap starting.
I20260927 10:53:34.708353 19699 tablet_bootstrap.cc:492] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec: Bootstrap starting.
I20260927 10:53:34.709587 19697 tablet_bootstrap.cc:654] T e27ae90b301745d5b6427ade46442dd8 P a578b8d774bf449f83cbb84f7979918a: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:34.710028 19699 tablet_bootstrap.cc:654] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:34.710345 19697 log.cc:824] T e27ae90b301745d5b6427ade46442dd8 P a578b8d774bf449f83cbb84f7979918a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:34.710745 19699 log.cc:824] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:34.711454 19698 tablet_bootstrap.cc:492] T e27ae90b301745d5b6427ade46442dd8 P 6dcafbebe4f64bb184e76ccdac26fd1b: Bootstrap starting.
I20260927 10:53:34.712276 19697 tablet_bootstrap.cc:492] T e27ae90b301745d5b6427ade46442dd8 P a578b8d774bf449f83cbb84f7979918a: No bootstrap required, opened a new log
I20260927 10:53:34.712478 19697 ts_tablet_manager.cc:1397] T e27ae90b301745d5b6427ade46442dd8 P a578b8d774bf449f83cbb84f7979918a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260927 10:53:34.712937 19699 tablet_bootstrap.cc:492] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec: No bootstrap required, opened a new log
I20260927 10:53:34.713106 19699 ts_tablet_manager.cc:1397] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:53:34.713495 19698 tablet_bootstrap.cc:654] T e27ae90b301745d5b6427ade46442dd8 P 6dcafbebe4f64bb184e76ccdac26fd1b: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:34.714284 19698 log.cc:824] T e27ae90b301745d5b6427ade46442dd8 P 6dcafbebe4f64bb184e76ccdac26fd1b: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:34.716374 19698 tablet_bootstrap.cc:492] T e27ae90b301745d5b6427ade46442dd8 P 6dcafbebe4f64bb184e76ccdac26fd1b: No bootstrap required, opened a new log
I20260927 10:53:34.716249 19697 raft_consensus.cc:348] T e27ae90b301745d5b6427ade46442dd8 P a578b8d774bf449f83cbb84f7979918a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcafbebe4f64bb184e76ccdac26fd1b" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 34995 } } peers { permanent_uuid: "d4c2254259c14c72a909512e134c94ec" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37713 } } peers { permanent_uuid: "a578b8d774bf449f83cbb84f7979918a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 44767 } }
I20260927 10:53:34.716480 19697 raft_consensus.cc:374] T e27ae90b301745d5b6427ade46442dd8 P a578b8d774bf449f83cbb84f7979918a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:34.716536 19697 raft_consensus.cc:729] T e27ae90b301745d5b6427ade46442dd8 P a578b8d774bf449f83cbb84f7979918a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a578b8d774bf449f83cbb84f7979918a, State: Initialized, Role: FOLLOWER
I20260927 10:53:34.716550 19698 ts_tablet_manager.cc:1397] T e27ae90b301745d5b6427ade46442dd8 P 6dcafbebe4f64bb184e76ccdac26fd1b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:53:34.717015 19697 consensus_queue.cc:260] T e27ae90b301745d5b6427ade46442dd8 P a578b8d774bf449f83cbb84f7979918a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcafbebe4f64bb184e76ccdac26fd1b" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 34995 } } peers { permanent_uuid: "d4c2254259c14c72a909512e134c94ec" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37713 } } peers { permanent_uuid: "a578b8d774bf449f83cbb84f7979918a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 44767 } }
I20260927 10:53:34.717432 19699 raft_consensus.cc:348] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcafbebe4f64bb184e76ccdac26fd1b" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 34995 } } peers { permanent_uuid: "d4c2254259c14c72a909512e134c94ec" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37713 } } peers { permanent_uuid: "a578b8d774bf449f83cbb84f7979918a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 44767 } }
I20260927 10:53:34.717689 19699 raft_consensus.cc:374] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:34.717743 19699 raft_consensus.cc:729] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4c2254259c14c72a909512e134c94ec, State: Initialized, Role: FOLLOWER
I20260927 10:53:34.717994 19697 ts_tablet_manager.cc:1428] T e27ae90b301745d5b6427ade46442dd8 P a578b8d774bf449f83cbb84f7979918a: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260927 10:53:34.718027 19680 heartbeater.cc:503] Master 127.14.68.62:33495 was elected leader, sending a full tablet report...
I20260927 10:53:34.718154 19699 consensus_queue.cc:260] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcafbebe4f64bb184e76ccdac26fd1b" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 34995 } } peers { permanent_uuid: "d4c2254259c14c72a909512e134c94ec" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37713 } } peers { permanent_uuid: "a578b8d774bf449f83cbb84f7979918a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 44767 } }
I20260927 10:53:34.719064 19699 ts_tablet_manager.cc:1428] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:53:34.719089 19549 heartbeater.cc:503] Master 127.14.68.62:33495 was elected leader, sending a full tablet report...
I20260927 10:53:34.720510 19698 raft_consensus.cc:348] T e27ae90b301745d5b6427ade46442dd8 P 6dcafbebe4f64bb184e76ccdac26fd1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcafbebe4f64bb184e76ccdac26fd1b" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 34995 } } peers { permanent_uuid: "d4c2254259c14c72a909512e134c94ec" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37713 } } peers { permanent_uuid: "a578b8d774bf449f83cbb84f7979918a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 44767 } }
I20260927 10:53:34.720769 19698 raft_consensus.cc:374] T e27ae90b301745d5b6427ade46442dd8 P 6dcafbebe4f64bb184e76ccdac26fd1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:34.720844 19698 raft_consensus.cc:729] T e27ae90b301745d5b6427ade46442dd8 P 6dcafbebe4f64bb184e76ccdac26fd1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dcafbebe4f64bb184e76ccdac26fd1b, State: Initialized, Role: FOLLOWER
I20260927 10:53:34.721254 19698 consensus_queue.cc:260] T e27ae90b301745d5b6427ade46442dd8 P 6dcafbebe4f64bb184e76ccdac26fd1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcafbebe4f64bb184e76ccdac26fd1b" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 34995 } } peers { permanent_uuid: "d4c2254259c14c72a909512e134c94ec" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37713 } } peers { permanent_uuid: "a578b8d774bf449f83cbb84f7979918a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 44767 } }
I20260927 10:53:34.722110 19698 ts_tablet_manager.cc:1428] T e27ae90b301745d5b6427ade46442dd8 P 6dcafbebe4f64bb184e76ccdac26fd1b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:53:34.722293 19418 heartbeater.cc:503] Master 127.14.68.62:33495 was elected leader, sending a full tablet report...
W20260927 10:53:34.742594 19550 tablet.cc:2369] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:53:34.766808 19704 raft_consensus.cc:482] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:34.767007 19704 raft_consensus.cc:504] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcafbebe4f64bb184e76ccdac26fd1b" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 34995 } } peers { permanent_uuid: "d4c2254259c14c72a909512e134c94ec" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37713 } } peers { permanent_uuid: "a578b8d774bf449f83cbb84f7979918a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 44767 } }
I20260927 10:53:34.768154 19704 leader_election.cc:288] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dcafbebe4f64bb184e76ccdac26fd1b (127.14.68.1:34995), a578b8d774bf449f83cbb84f7979918a (127.14.68.3:44767)
I20260927 10:53:34.770186 19551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.768366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:34995 (local address 127.14.68.2:50097)
0927 10:53:34.768529 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:34.768563 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:34.768669 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:34.769008 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:34.769011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:34.769025 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:34.769539 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:53:34.769541 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:34.769552 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:53:34.769602 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:34.769893 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:34.769896 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:34.769899 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 10:53:34.769998 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 10:53:34.770034 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 10:53:34.770260 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.768503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:50097 (local address 127.14.68.1:34995)
0927 10:53:34.768644 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:34.768649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:34.768673 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:53:34.768785 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:34.768788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:34.768836 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:34.768924 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:34.769743 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:34.769748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:34.769752 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:34.769766 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:34.769790 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:34.769803 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:34.769896 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:34.770006 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 10:53:34.770051 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:53:34.770411 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.768669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:44767 (local address 127.14.68.2:37867)
0927 10:53:34.769058 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:34.769068 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:34.769167 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:34.769403 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:34.769406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:34.769419 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:34.769831 (+ 412us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:53:34.769835 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:34.769848 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:53:34.769877 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:34.770145 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:34.770148 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:34.770151 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 10:53:34.770228 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 10:53:34.770261 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":163,"threads_started":1}
I20260927 10:53:34.770431 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.768810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:37867 (local address 127.14.68.3:44767)
0927 10:53:34.768945 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:34.768949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:34.769142 (+ 193us) server_negotiation.cc:407] Connection header received
0927 10:53:34.769247 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:34.769250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:34.769292 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:34.769388 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:34.769994 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:34.769997 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:34.769999 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:34.770014 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:34.770034 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:34.770044 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:34.770140 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:34.770248 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 10:53:34.770289 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:53:34.770850 19372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27ae90b301745d5b6427ade46442dd8" candidate_uuid: "d4c2254259c14c72a909512e134c94ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dcafbebe4f64bb184e76ccdac26fd1b" is_pre_election: true
I20260927 10:53:34.771102 19634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27ae90b301745d5b6427ade46442dd8" candidate_uuid: "d4c2254259c14c72a909512e134c94ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a578b8d774bf449f83cbb84f7979918a" is_pre_election: true
I20260927 10:53:34.771190 19372 raft_consensus.cc:2454] T e27ae90b301745d5b6427ade46442dd8 P 6dcafbebe4f64bb184e76ccdac26fd1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4c2254259c14c72a909512e134c94ec in term 0.
I20260927 10:53:34.771413 19634 raft_consensus.cc:2454] T e27ae90b301745d5b6427ade46442dd8 P a578b8d774bf449f83cbb84f7979918a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4c2254259c14c72a909512e134c94ec in term 0.
I20260927 10:53:34.771665 19438 leader_election.cc:302] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dcafbebe4f64bb184e76ccdac26fd1b, d4c2254259c14c72a909512e134c94ec; no voters:
I20260927 10:53:34.771978 19704 raft_consensus.cc:2781] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:34.772095 19704 raft_consensus.cc:482] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:34.772141 19704 raft_consensus.cc:3037] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:34.773339 19704 raft_consensus.cc:504] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcafbebe4f64bb184e76ccdac26fd1b" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 34995 } } peers { permanent_uuid: "d4c2254259c14c72a909512e134c94ec" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37713 } } peers { permanent_uuid: "a578b8d774bf449f83cbb84f7979918a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 44767 } }
I20260927 10:53:34.773638 19704 leader_election.cc:288] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec [CANDIDATE]: Term 1 election: Requested vote from peers 6dcafbebe4f64bb184e76ccdac26fd1b (127.14.68.1:34995), a578b8d774bf449f83cbb84f7979918a (127.14.68.3:44767)
I20260927 10:53:34.773984 19372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27ae90b301745d5b6427ade46442dd8" candidate_uuid: "d4c2254259c14c72a909512e134c94ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dcafbebe4f64bb184e76ccdac26fd1b"
I20260927 10:53:34.774106 19634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27ae90b301745d5b6427ade46442dd8" candidate_uuid: "d4c2254259c14c72a909512e134c94ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a578b8d774bf449f83cbb84f7979918a"
I20260927 10:53:34.774196 19372 raft_consensus.cc:3037] T e27ae90b301745d5b6427ade46442dd8 P 6dcafbebe4f64bb184e76ccdac26fd1b [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:34.774237 19634 raft_consensus.cc:3037] T e27ae90b301745d5b6427ade46442dd8 P a578b8d774bf449f83cbb84f7979918a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:34.775219 19372 raft_consensus.cc:2454] T e27ae90b301745d5b6427ade46442dd8 P 6dcafbebe4f64bb184e76ccdac26fd1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4c2254259c14c72a909512e134c94ec in term 1.
I20260927 10:53:34.775221 19634 raft_consensus.cc:2454] T e27ae90b301745d5b6427ade46442dd8 P a578b8d774bf449f83cbb84f7979918a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4c2254259c14c72a909512e134c94ec in term 1.
I20260927 10:53:34.775488 19438 leader_election.cc:302] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dcafbebe4f64bb184e76ccdac26fd1b, d4c2254259c14c72a909512e134c94ec; no voters:
I20260927 10:53:34.775660 19704 raft_consensus.cc:2781] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:34.775908 19704 raft_consensus.cc:686] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec [term 1 LEADER]: Becoming Leader. State: Replica: d4c2254259c14c72a909512e134c94ec, State: Running, Role: LEADER
I20260927 10:53:34.776209 19704 consensus_queue.cc:237] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcafbebe4f64bb184e76ccdac26fd1b" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 34995 } } peers { permanent_uuid: "d4c2254259c14c72a909512e134c94ec" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37713 } } peers { permanent_uuid: "a578b8d774bf449f83cbb84f7979918a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 44767 } }
I20260927 10:53:34.779247 19229 catalog_manager.cc:5495] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec reported cstate change: term changed from 0 to 1, leader changed from <none> to d4c2254259c14c72a909512e134c94ec (127.14.68.2). New cstate: current_term: 1 leader_uuid: "d4c2254259c14c72a909512e134c94ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcafbebe4f64bb184e76ccdac26fd1b" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 34995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4c2254259c14c72a909512e134c94ec" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 37713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a578b8d774bf449f83cbb84f7979918a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 44767 } health_report { overall_health: UNKNOWN } } }
I20260927 10:53:34.797257 19694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.795606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58480 (local address 127.14.68.2:37713)
0927 10:53:34.795772 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:34.795777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:34.795792 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:53:34.795959 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:34.795961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:34.796021 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:34.796113 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:34.796840 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:34.796844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:34.796846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:34.796858 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:34.796879 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:34.796891 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:34.796990 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:34.797066 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 10:53:34.797103 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:53:34.803040 19634 raft_consensus.cc:1261] T e27ae90b301745d5b6427ade46442dd8 P a578b8d774bf449f83cbb84f7979918a [term 1 FOLLOWER]: Refusing update from remote peer d4c2254259c14c72a909512e134c94ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:34.803040 19372 raft_consensus.cc:1261] T e27ae90b301745d5b6427ade46442dd8 P 6dcafbebe4f64bb184e76ccdac26fd1b [term 1 FOLLOWER]: Refusing update from remote peer d4c2254259c14c72a909512e134c94ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:34.803783 19707 consensus_queue.cc:1035] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a578b8d774bf449f83cbb84f7979918a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 44767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:34.803972 19704 consensus_queue.cc:1035] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dcafbebe4f64bb184e76ccdac26fd1b" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 34995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:34.811812 19711 mvcc.cc:205] Tried to move back new op lower bound from 7333914275026063360 to 7333914274925617152. Current Snapshot: MvccSnapshot[applied={T|T < 7333914275026063360}]
I20260927 10:53:34.823568 19229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:34.823796 19229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:34.826376 19712 mvcc.cc:205] Tried to move back new op lower bound from 7333914275026063360 to 7333914274925617152. Current Snapshot: MvccSnapshot[applied={T|T < 7333914275026063360}]
I20260927 10:53:34.827611 19229 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81b8baffe4a24759b6c00f28ad6b3a86: Sending DeleteTablet for 3 replicas of tablet e27ae90b301745d5b6427ade46442dd8
I20260927 10:53:34.828405 19614 tablet_service.cc:1514] Processing DeleteTablet for tablet e27ae90b301745d5b6427ade46442dd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:34 UTC) from {username='slave'} at 127.0.0.1:46694
I20260927 10:53:34.828708 19352 tablet_service.cc:1514] Processing DeleteTablet for tablet e27ae90b301745d5b6427ade46442dd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:34 UTC) from {username='slave'} at 127.0.0.1:51590
I20260927 10:53:34.828826 19613 tablet_service.cc:1514] Processing DeleteTablet for tablet e27ae90b301745d5b6427ade46442dd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:34 UTC) from {username='slave'} at 127.0.0.1:46694
I20260927 10:53:34.829033 19483 tablet_service.cc:1514] Processing DeleteTablet for tablet e27ae90b301745d5b6427ade46442dd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:34 UTC) from {username='slave'} at 127.0.0.1:58474
I20260927 10:53:34.829552 19721 tablet_replica.cc:330] T e27ae90b301745d5b6427ade46442dd8 P 6dcafbebe4f64bb184e76ccdac26fd1b: stopping tablet replica
I20260927 10:53:34.829725 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 19290
I20260927 10:53:34.829962 19721 raft_consensus.cc:2229] T e27ae90b301745d5b6427ade46442dd8 P 6dcafbebe4f64bb184e76ccdac26fd1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:34.830318 19720 tablet_replica.cc:330] T e27ae90b301745d5b6427ade46442dd8 P a578b8d774bf449f83cbb84f7979918a: stopping tablet replica
I20260927 10:53:34.830667 19720 raft_consensus.cc:2229] T e27ae90b301745d5b6427ade46442dd8 P a578b8d774bf449f83cbb84f7979918a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:34.830962 19720 raft_consensus.cc:2258] T e27ae90b301745d5b6427ade46442dd8 P a578b8d774bf449f83cbb84f7979918a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:34.831321 19722 tablet_replica.cc:330] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec: stopping tablet replica
I20260927 10:53:34.831568 19722 raft_consensus.cc:2229] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:53:34.831931 19722 raft_consensus.cc:2258] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:34.833290 19720 ts_tablet_manager.cc:1905] T e27ae90b301745d5b6427ade46442dd8 P a578b8d774bf449f83cbb84f7979918a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:34.836660 19720 ts_tablet_manager.cc:1918] T e27ae90b301745d5b6427ade46442dd8 P a578b8d774bf449f83cbb84f7979918a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:34.836767 19720 log.cc:1196] T e27ae90b301745d5b6427ade46442dd8 P a578b8d774bf449f83cbb84f7979918a: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/e27ae90b301745d5b6427ade46442dd8
I20260927 10:53:34.837069 19722 ts_tablet_manager.cc:1905] T e27ae90b301745d5b6427ade46442dd8 P d4c2254259c14c72a909512e134c94ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:34.837105 19720 ts_tablet_manager.cc:1939] T e27ae90b301745d5b6427ade46442dd8 P a578b8d774bf449f83cbb84f7979918a: Deleting consensus metadata
I20260927 10:53:34.838433 19216 catalog_manager.cc:4841] TS a578b8d774bf449f83cbb84f7979918a (127.14.68.3:44767): tablet e27ae90b301745d5b6427ade46442dd8 (table test-table [id=40bbd22bebf04bcc89b2602815ae811e]) successfully deleted
W20260927 10:53:34.838922 19216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.68.1:34995 (error 108)
I20260927 10:53:34.839224 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 19421
W20260927 10:53:34.840921 19216 catalog_manager.cc:4808] TS a578b8d774bf449f83cbb84f7979918a (127.14.68.3:44767): delete failed for tablet e27ae90b301745d5b6427ade46442dd8 because the tablet was not found. No further retry: Not found: Tablet not found: e27ae90b301745d5b6427ade46442dd8
I20260927 10:53:34.841584 19692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.841309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:34995 (local address 127.0.0.1:51600)
0927 10:53:34.841444 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:34.841510 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.1:34995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 10:53:34.841750 19216 catalog_manager.cc:4584] TS 6dcafbebe4f64bb184e76ccdac26fd1b (127.14.68.1:34995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e27ae90b301745d5b6427ade46442dd8: Network error: Client connection negotiation failed: client connection to 127.14.68.1:34995: connect: Connection refused (error 111)
I20260927 10:53:34.845594 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 19552
I20260927 10:53:34.846511 19692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:34.845811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:37713 (local address 127.0.0.1:58488)
0927 10:53:34.846345 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:34.846411 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.2:37713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427}
I20260927 10:53:34.851451 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 19199
2026-09-27T10:53:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (986 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 10:53:34.872888 14608 mini_oidc.cc:152] Starting JWKS server
I20260927 10:53:34.874482 14608 webserver.cc:460] Webserver started at https://127.0.0.1:43019/ using document root <none> and password file <none>
I20260927 10:53:34.874672 14608 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 10:53:34.874950 14608 webserver.cc:460] Webserver started at http://127.0.0.1:40143/ using document root <none> and password file <none>
2026-09-27T10:53:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:34Z Disabled control of system clock
I20260927 10:53:34.883594 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:38699
--webserver_interface=127.14.68.62
--webserver_port=0
--builtin_ntp_servers=127.14.68.20:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:38699
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43019/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506389460539-14608-0/testchainca.cert with env {}
W20260927 10:53:34.991907 19729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:34.992312 19729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:34.992372 19729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:34.996331 19729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:34.996385 19729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:34.996402 19729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:34.996418 19729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:34.996433 19729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 10:53:34.996449 19729 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 10:53:34.996462 19729 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43019/jwks.json
I20260927 10:53:35.001858 19729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:38699
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:38699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43019/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506389460539-14608-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:35.002897 19729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:35.003902 19729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:35.012259 19735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:35.012247 19734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:35.012249 19737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:35.012454 19729 server_base.cc:1034] running on GCE node
I20260927 10:53:35.013024 19729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:35.014118 19729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:35.015281 19729 hybrid_clock.cc:648] HybridClock initialized: now 1790506415015260 us; error 32 us; skew 500 ppm
I20260927 10:53:35.017145 19729 webserver.cc:460] Webserver started at http://127.14.68.62:35241/ using document root <none> and password file <none>
I20260927 10:53:35.017673 19729 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:35.017727 19729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:35.017877 19729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:35.019914 19729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "70d25fe48890466ab642846b644326ef"
format_stamp: "Formatted at 2026-09-27 10:53:35 on dist-test-slave-tz51"
server_key: "18aa545dc56a5eb8e56e1b81db4774d0"
server_key_iv: "8ec40d2d9ec2f0c5c69abb3fd1c6e312"
server_key_version: "encryptionkey@0"
I20260927 10:53:35.020419 19729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "70d25fe48890466ab642846b644326ef"
format_stamp: "Formatted at 2026-09-27 10:53:35 on dist-test-slave-tz51"
server_key: "18aa545dc56a5eb8e56e1b81db4774d0"
server_key_iv: "8ec40d2d9ec2f0c5c69abb3fd1c6e312"
server_key_version: "encryptionkey@0"
I20260927 10:53:35.023846 19729 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:53:35.026202 19743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:35.027441 19729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:35.027542 19729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "70d25fe48890466ab642846b644326ef"
format_stamp: "Formatted at 2026-09-27 10:53:35 on dist-test-slave-tz51"
server_key: "18aa545dc56a5eb8e56e1b81db4774d0"
server_key_iv: "8ec40d2d9ec2f0c5c69abb3fd1c6e312"
server_key_version: "encryptionkey@0"
I20260927 10:53:35.027626 19729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:35.042944 19729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:35.043540 19729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:35.043680 19729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:35.050570 19729 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:38699
I20260927 10:53:35.050576 19795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:38699 every 8 connection(s)
I20260927 10:53:35.051507 19729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:35.054665 19796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:35.058846 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 19729
I20260927 10:53:35.058952 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:53:35.059245 14608 external_mini_cluster.cc:1428] Setting key 32807e77ef407492cf4431abf16d5efa
I20260927 10:53:35.060252 19796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d25fe48890466ab642846b644326ef: Bootstrap starting.
I20260927 10:53:35.062371 19796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70d25fe48890466ab642846b644326ef: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:35.063102 19796 log.cc:824] T 00000000000000000000000000000000 P 70d25fe48890466ab642846b644326ef: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:35.064924 19796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d25fe48890466ab642846b644326ef: No bootstrap required, opened a new log
I20260927 10:53:35.068168 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:35.060613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40512 (local address 127.14.68.62:38699)
0927 10:53:35.061198 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:35.061212 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:35.061249 (+ 37us) server_negotiation.cc:407] Connection header received
0927 10:53:35.061847 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:35.061863 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:35.062242 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:35.062569 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:35.063012 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:35.064040 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:35.064926 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:35.065447 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:35.065616 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:35.066481 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:35.066485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:35.066497 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:35.066864 (+ 367us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:35.066914 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:35.066933 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:35.067037 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:35.067189 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 10:53:35.067426 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":125,"threads_started":1}
I20260927 10:53:35.068652 19796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70d25fe48890466ab642846b644326ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d25fe48890466ab642846b644326ef" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38699 } }
I20260927 10:53:35.068876 19796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70d25fe48890466ab642846b644326ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:35.068938 19796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70d25fe48890466ab642846b644326ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d25fe48890466ab642846b644326ef, State: Initialized, Role: FOLLOWER
I20260927 10:53:35.069342 19796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70d25fe48890466ab642846b644326ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d25fe48890466ab642846b644326ef" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38699 } }
I20260927 10:53:35.069499 19796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70d25fe48890466ab642846b644326ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:35.069581 19796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70d25fe48890466ab642846b644326ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:35.069698 19796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70d25fe48890466ab642846b644326ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:35.070640 19796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70d25fe48890466ab642846b644326ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d25fe48890466ab642846b644326ef" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38699 } }
I20260927 10:53:35.070906 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:53:35.070991 19796 leader_election.cc:302] T 00000000000000000000000000000000 P 70d25fe48890466ab642846b644326ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70d25fe48890466ab642846b644326ef; no voters:
I20260927 10:53:35.071324 19796 leader_election.cc:288] T 00000000000000000000000000000000 P 70d25fe48890466ab642846b644326ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:35.071403 19801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70d25fe48890466ab642846b644326ef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:35.071635 19801 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70d25fe48890466ab642846b644326ef [term 1 LEADER]: Becoming Leader. State: Replica: 70d25fe48890466ab642846b644326ef, State: Running, Role: LEADER
I20260927 10:53:35.071950 19801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70d25fe48890466ab642846b644326ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d25fe48890466ab642846b644326ef" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38699 } }
I20260927 10:53:35.072494 19796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70d25fe48890466ab642846b644326ef [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:35.073580 19803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d25fe48890466ab642846b644326ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70d25fe48890466ab642846b644326ef. Latest consensus state: current_term: 1 leader_uuid: "70d25fe48890466ab642846b644326ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d25fe48890466ab642846b644326ef" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38699 } } }
I20260927 10:53:35.073737 19803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d25fe48890466ab642846b644326ef [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:35.073580 19802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d25fe48890466ab642846b644326ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70d25fe48890466ab642846b644326ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d25fe48890466ab642846b644326ef" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38699 } } }
I20260927 10:53:35.073822 19802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d25fe48890466ab642846b644326ef [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:35.074218 19810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:35.076903 19810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:35.081732 19810 catalog_manager.cc:1347] Generated new cluster ID: c6979234371c44488b2546f32a16de88
I20260927 10:53:35.081791 19810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:35.089548 19810 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:35.090430 19810 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:35.098536 19810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70d25fe48890466ab642846b644326ef: Generated new TSK 0
I20260927 10:53:35.099184 19810 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 10:53:35.127624 19826 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 10:53:35.127782 19826 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 10:53:35.128178 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:35.111570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40520 (local address 127.14.68.62:38699)
0927 10:53:35.111714 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:35.111719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:35.111890 (+ 171us) server_negotiation.cc:407] Connection header received
0927 10:53:35.112064 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:35.112067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:35.112119 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:35.112230 (+ 111us) server_negotiation.cc:226] Negotiated authn=JWT
0927 10:53:35.112451 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:35.113261 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:35.114087 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:35.114484 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:35.114587 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:35.114874 (+ 287us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 10:53:35.127805 (+ 12931us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43019/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 10:53:35.128036 (+ 231us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40520: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43019/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 10:53:35.128254 19799 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40520: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43019/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 10:53:35.129391 19825 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:35.111454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:38699 (local address 127.0.0.1:40520)
0927 10:53:35.111813 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:35.111825 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:35.111923 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:35.112259 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:35.112261 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:35.112268 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0927 10:53:35.112355 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:35.112360 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:35.113364 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:35.113366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:35.113979 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:35.113986 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:35.114599 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:35.114601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:35.114664 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:35.114705 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 10:53:35.128005 (+ 13300us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43019/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 10:53:35.128183 (+ 178us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.68.62:38699: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43019/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":108,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.14.68.62:38699: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43019/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 10:53:35.130334 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 19729
2026-09-27T10:53:35Z chronyd exiting
I20260927 10:53:35.167533 14608 test_util.cc:175] -----------------------------------------------
I20260927 10:53:35.167618 14608 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506389460539-14608-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (296 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 10:53:35.168659 14608 mini_oidc.cc:152] Starting JWKS server
I20260927 10:53:35.169474 14608 webserver.cc:460] Webserver started at https://127.0.0.1:39449/ using document root <none> and password file <none>
I20260927 10:53:35.169551 14608 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 10:53:35.169749 14608 webserver.cc:460] Webserver started at http://127.0.0.1:36401/ using document root <none> and password file <none>
2026-09-27T10:53:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:35Z Disabled control of system clock
I20260927 10:53:35.178674 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:34929
--webserver_interface=127.14.68.62
--webserver_port=0
--builtin_ntp_servers=127.14.68.20:36777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:34929
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39449/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506389460539-14608-0/testchainca.cert with env {}
W20260927 10:53:35.287894 19832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:35.288177 19832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:35.288216 19832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:35.292194 19832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:35.292249 19832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:35.292265 19832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:35.292281 19832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:35.292297 19832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 10:53:35.292312 19832 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 10:53:35.292326 19832 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39449/jwks.json
I20260927 10:53:35.297698 19832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:36777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:34929
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:34929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39449/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506389460539-14608-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:35.298696 19832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:35.299636 19832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:35.307971 19840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:35.308154 19832 server_base.cc:1034] running on GCE node
W20260927 10:53:35.307981 19838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:35.308320 19837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:35.308941 19832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:35.310043 19832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:35.311203 19832 hybrid_clock.cc:648] HybridClock initialized: now 1790506415311171 us; error 45 us; skew 500 ppm
I20260927 10:53:35.313067 19832 webserver.cc:460] Webserver started at http://127.14.68.62:43653/ using document root <none> and password file <none>
I20260927 10:53:35.313699 19832 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:35.313769 19832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:35.314018 19832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:35.316118 19832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "1017a6e358cb4edabb2b325819c638a4"
format_stamp: "Formatted at 2026-09-27 10:53:35 on dist-test-slave-tz51"
server_key: "b4b3a352ee91ad23bf860fb66b01c398"
server_key_iv: "e05f5ac5ea9a6396c3fe26dfcbc3e874"
server_key_version: "encryptionkey@0"
I20260927 10:53:35.316744 19832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "1017a6e358cb4edabb2b325819c638a4"
format_stamp: "Formatted at 2026-09-27 10:53:35 on dist-test-slave-tz51"
server_key: "b4b3a352ee91ad23bf860fb66b01c398"
server_key_iv: "e05f5ac5ea9a6396c3fe26dfcbc3e874"
server_key_version: "encryptionkey@0"
I20260927 10:53:35.320353 19832 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:53:35.322701 19846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:35.324029 19832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 10:53:35.324185 19832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "1017a6e358cb4edabb2b325819c638a4"
format_stamp: "Formatted at 2026-09-27 10:53:35 on dist-test-slave-tz51"
server_key: "b4b3a352ee91ad23bf860fb66b01c398"
server_key_iv: "e05f5ac5ea9a6396c3fe26dfcbc3e874"
server_key_version: "encryptionkey@0"
I20260927 10:53:35.324295 19832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:35.343662 19832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:35.344403 19832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:35.344614 19832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:35.351585 19832 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:34929
I20260927 10:53:35.351598 19898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:34929 every 8 connection(s)
I20260927 10:53:35.352536 19832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:35.354292 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 19832
I20260927 10:53:35.354383 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:53:35.354663 14608 external_mini_cluster.cc:1428] Setting key 9e998978c4bb870995ac259c412be9b2
I20260927 10:53:35.355966 19899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:35.362480 19899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1017a6e358cb4edabb2b325819c638a4: Bootstrap starting.
I20260927 10:53:35.364697 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:35.356040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34112 (local address 127.14.68.62:34929)
0927 10:53:35.356711 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:35.356731 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:35.356772 (+ 41us) server_negotiation.cc:407] Connection header received
0927 10:53:35.357629 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:35.357651 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:35.358110 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:35.358483 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:35.358900 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:35.360324 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:35.361114 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:35.361780 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:35.361926 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:35.362759 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:35.362765 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:35.362778 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:35.363256 (+ 478us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:35.363318 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:35.363345 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:35.363438 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:35.363535 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 10:53:35.363832 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":115,"threads_started":1}
I20260927 10:53:35.365094 19899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1017a6e358cb4edabb2b325819c638a4: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:35.365801 19899 log.cc:824] T 00000000000000000000000000000000 P 1017a6e358cb4edabb2b325819c638a4: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:35.367413 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:53:35.367883 19899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1017a6e358cb4edabb2b325819c638a4: No bootstrap required, opened a new log
I20260927 10:53:35.371424 19899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1017a6e358cb4edabb2b325819c638a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1017a6e358cb4edabb2b325819c638a4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 34929 } }
I20260927 10:53:35.371661 19899 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1017a6e358cb4edabb2b325819c638a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:35.371757 19899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1017a6e358cb4edabb2b325819c638a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1017a6e358cb4edabb2b325819c638a4, State: Initialized, Role: FOLLOWER
I20260927 10:53:35.372213 19899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1017a6e358cb4edabb2b325819c638a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1017a6e358cb4edabb2b325819c638a4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 34929 } }
I20260927 10:53:35.372359 19899 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1017a6e358cb4edabb2b325819c638a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:35.372428 19899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1017a6e358cb4edabb2b325819c638a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:35.372524 19899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1017a6e358cb4edabb2b325819c638a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:35.373428 19899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1017a6e358cb4edabb2b325819c638a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1017a6e358cb4edabb2b325819c638a4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 34929 } }
I20260927 10:53:35.373765 19899 leader_election.cc:302] T 00000000000000000000000000000000 P 1017a6e358cb4edabb2b325819c638a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1017a6e358cb4edabb2b325819c638a4; no voters:
I20260927 10:53:35.374073 19899 leader_election.cc:288] T 00000000000000000000000000000000 P 1017a6e358cb4edabb2b325819c638a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:35.374186 19904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1017a6e358cb4edabb2b325819c638a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:35.374408 19904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1017a6e358cb4edabb2b325819c638a4 [term 1 LEADER]: Becoming Leader. State: Replica: 1017a6e358cb4edabb2b325819c638a4, State: Running, Role: LEADER
I20260927 10:53:35.374716 19904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1017a6e358cb4edabb2b325819c638a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1017a6e358cb4edabb2b325819c638a4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 34929 } }
I20260927 10:53:35.375322 19899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1017a6e358cb4edabb2b325819c638a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:35.376472 19905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1017a6e358cb4edabb2b325819c638a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1017a6e358cb4edabb2b325819c638a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1017a6e358cb4edabb2b325819c638a4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 34929 } } }
I20260927 10:53:35.376609 19905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1017a6e358cb4edabb2b325819c638a4 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:35.376921 19906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1017a6e358cb4edabb2b325819c638a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1017a6e358cb4edabb2b325819c638a4. Latest consensus state: current_term: 1 leader_uuid: "1017a6e358cb4edabb2b325819c638a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1017a6e358cb4edabb2b325819c638a4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 34929 } } }
I20260927 10:53:35.377018 19906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1017a6e358cb4edabb2b325819c638a4 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:35.377104 19913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:35.380510 19913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:35.385499 19913 catalog_manager.cc:1347] Generated new cluster ID: 7f768f443fb043b4a84ab5ef38e956e3
I20260927 10:53:35.385572 19913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:35.411571 19913 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:35.412801 19913 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:35.418722 19913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1017a6e358cb4edabb2b325819c638a4: Generated new TSK 0
I20260927 10:53:35.419478 19913 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 10:53:35.443622 19929 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 10:53:35.443746 19929 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 10:53:35.444463 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:35.428561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34116 (local address 127.14.68.62:34929)
0927 10:53:35.428696 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:35.428701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:35.428888 (+ 187us) server_negotiation.cc:407] Connection header received
0927 10:53:35.429071 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:35.429074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:35.429130 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:35.429240 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0927 10:53:35.429452 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:35.430255 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:35.431064 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:35.431413 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:35.431513 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:35.431797 (+ 284us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 10:53:35.444054 (+ 12257us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39449/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 10:53:35.444322 (+ 268us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34116: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39449/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 10:53:35.444532 19902 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34116: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39449/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 10:53:35.444562 19928 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:35.428447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:34929 (local address 127.0.0.1:34116)
0927 10:53:35.428820 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:35.428833 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:35.428919 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:35.429251 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:35.429253 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:35.429259 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0927 10:53:35.429350 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:35.429355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:35.430359 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:35.430361 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:35.430940 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:35.430946 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:35.431532 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:35.431534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:35.431600 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:35.431619 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 10:53:35.444254 (+ 12635us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39449/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 10:53:35.444397 (+ 143us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.68.62:34929: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39449/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":106,"threads_started":1}
I20260927 10:53:35.445163 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 19832
2026-09-27T10:53:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (315 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 10:53:35.484052 14608 mini_oidc.cc:152] Starting JWKS server
I20260927 10:53:35.484937 14608 webserver.cc:460] Webserver started at https://127.0.0.1:37083/ using document root <none> and password file <none>
I20260927 10:53:35.485049 14608 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 10:53:35.485339 14608 webserver.cc:460] Webserver started at http://127.0.0.1:39965/ using document root <none> and password file <none>
2026-09-27T10:53:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:35Z Disabled control of system clock
I20260927 10:53:35.493988 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:45617
--webserver_interface=127.14.68.62
--webserver_port=0
--builtin_ntp_servers=127.14.68.20:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:45617
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37083/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 10:53:35.602217 19935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:35.602504 19935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:35.602571 19935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:35.606590 19935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:35.606675 19935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:35.606704 19935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:35.606739 19935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:35.606801 19935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 10:53:35.606840 19935 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 10:53:35.606873 19935 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37083/jwks.json
I20260927 10:53:35.611563 19935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:45617
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:45617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37083/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:35.612622 19935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:35.613951 19935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:53:35.622380 19935 server_base.cc:1034] running on GCE node
W20260927 10:53:35.622351 19943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:35.622428 19940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:35.622391 19941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:35.623068 19935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:35.624260 19935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:35.625423 19935 hybrid_clock.cc:648] HybridClock initialized: now 1790506415625395 us; error 40 us; skew 500 ppm
I20260927 10:53:35.627308 19935 webserver.cc:460] Webserver started at http://127.14.68.62:33169/ using document root <none> and password file <none>
I20260927 10:53:35.627951 19935 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:35.628014 19935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:35.628284 19935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:35.630316 19935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "bba6ece1b4b74bfea3c55d7c6f6799c0"
format_stamp: "Formatted at 2026-09-27 10:53:35 on dist-test-slave-tz51"
server_key: "670c57f1e4a4134992b2df00d8c6bd3a"
server_key_iv: "aa46d67552ad7329f0dd6c3620e0c945"
server_key_version: "encryptionkey@0"
I20260927 10:53:35.630831 19935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "bba6ece1b4b74bfea3c55d7c6f6799c0"
format_stamp: "Formatted at 2026-09-27 10:53:35 on dist-test-slave-tz51"
server_key: "670c57f1e4a4134992b2df00d8c6bd3a"
server_key_iv: "aa46d67552ad7329f0dd6c3620e0c945"
server_key_version: "encryptionkey@0"
I20260927 10:53:35.634457 19935 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:53:35.636821 19949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:35.638121 19935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:53:35.638263 19935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "bba6ece1b4b74bfea3c55d7c6f6799c0"
format_stamp: "Formatted at 2026-09-27 10:53:35 on dist-test-slave-tz51"
server_key: "670c57f1e4a4134992b2df00d8c6bd3a"
server_key_iv: "aa46d67552ad7329f0dd6c3620e0c945"
server_key_version: "encryptionkey@0"
I20260927 10:53:35.638370 19935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:35.659794 19935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:35.660490 19935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:35.660645 19935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:35.667456 19935 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:45617
I20260927 10:53:35.667488 20001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:45617 every 8 connection(s)
I20260927 10:53:35.668476 19935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:35.669188 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 19935
I20260927 10:53:35.669305 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:53:35.669575 14608 external_mini_cluster.cc:1428] Setting key 4d267ddbce8e3963b898f52af2ec9710
I20260927 10:53:35.671923 20002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:35.677399 20002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bba6ece1b4b74bfea3c55d7c6f6799c0: Bootstrap starting.
I20260927 10:53:35.679098 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:35.670839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51098 (local address 127.14.68.62:45617)
0927 10:53:35.671448 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:35.671464 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:35.672012 (+ 548us) server_negotiation.cc:407] Connection header received
0927 10:53:35.672756 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:35.672772 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:35.673188 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:35.673528 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:35.673950 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:35.674908 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:35.675863 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:35.676338 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:35.676515 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:35.677394 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:35.677399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:35.677412 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:35.677846 (+ 434us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:35.677901 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:35.677922 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:35.678024 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:35.678157 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 10:53:35.678412 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":140,"threads_started":1}
I20260927 10:53:35.679883 20002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bba6ece1b4b74bfea3c55d7c6f6799c0: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:35.680635 20002 log.cc:824] T 00000000000000000000000000000000 P bba6ece1b4b74bfea3c55d7c6f6799c0: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:35.682058 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:53:35.682823 20002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bba6ece1b4b74bfea3c55d7c6f6799c0: No bootstrap required, opened a new log
I20260927 10:53:35.686546 20002 raft_consensus.cc:348] T 00000000000000000000000000000000 P bba6ece1b4b74bfea3c55d7c6f6799c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba6ece1b4b74bfea3c55d7c6f6799c0" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45617 } }
I20260927 10:53:35.686760 20002 raft_consensus.cc:374] T 00000000000000000000000000000000 P bba6ece1b4b74bfea3c55d7c6f6799c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:35.686839 20002 raft_consensus.cc:729] T 00000000000000000000000000000000 P bba6ece1b4b74bfea3c55d7c6f6799c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba6ece1b4b74bfea3c55d7c6f6799c0, State: Initialized, Role: FOLLOWER
I20260927 10:53:35.687230 20002 consensus_queue.cc:260] T 00000000000000000000000000000000 P bba6ece1b4b74bfea3c55d7c6f6799c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba6ece1b4b74bfea3c55d7c6f6799c0" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45617 } }
I20260927 10:53:35.687371 20002 raft_consensus.cc:388] T 00000000000000000000000000000000 P bba6ece1b4b74bfea3c55d7c6f6799c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:35.687436 20002 raft_consensus.cc:482] T 00000000000000000000000000000000 P bba6ece1b4b74bfea3c55d7c6f6799c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:35.687563 20002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bba6ece1b4b74bfea3c55d7c6f6799c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:35.688537 20002 raft_consensus.cc:504] T 00000000000000000000000000000000 P bba6ece1b4b74bfea3c55d7c6f6799c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba6ece1b4b74bfea3c55d7c6f6799c0" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45617 } }
I20260927 10:53:35.688875 20002 leader_election.cc:302] T 00000000000000000000000000000000 P bba6ece1b4b74bfea3c55d7c6f6799c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bba6ece1b4b74bfea3c55d7c6f6799c0; no voters:
I20260927 10:53:35.689232 20002 leader_election.cc:288] T 00000000000000000000000000000000 P bba6ece1b4b74bfea3c55d7c6f6799c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:35.689307 20007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bba6ece1b4b74bfea3c55d7c6f6799c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:35.689560 20007 raft_consensus.cc:686] T 00000000000000000000000000000000 P bba6ece1b4b74bfea3c55d7c6f6799c0 [term 1 LEADER]: Becoming Leader. State: Replica: bba6ece1b4b74bfea3c55d7c6f6799c0, State: Running, Role: LEADER
I20260927 10:53:35.689854 20007 consensus_queue.cc:237] T 00000000000000000000000000000000 P bba6ece1b4b74bfea3c55d7c6f6799c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba6ece1b4b74bfea3c55d7c6f6799c0" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45617 } }
I20260927 10:53:35.690429 20002 sys_catalog.cc:564] T 00000000000000000000000000000000 P bba6ece1b4b74bfea3c55d7c6f6799c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:35.691421 20008 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba6ece1b4b74bfea3c55d7c6f6799c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bba6ece1b4b74bfea3c55d7c6f6799c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba6ece1b4b74bfea3c55d7c6f6799c0" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45617 } } }
I20260927 10:53:35.691568 20008 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba6ece1b4b74bfea3c55d7c6f6799c0 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:35.691867 20009 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba6ece1b4b74bfea3c55d7c6f6799c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bba6ece1b4b74bfea3c55d7c6f6799c0. Latest consensus state: current_term: 1 leader_uuid: "bba6ece1b4b74bfea3c55d7c6f6799c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba6ece1b4b74bfea3c55d7c6f6799c0" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45617 } } }
I20260927 10:53:35.691969 20009 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba6ece1b4b74bfea3c55d7c6f6799c0 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:35.691952 20016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:35.695155 20016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:35.700672 20016 catalog_manager.cc:1347] Generated new cluster ID: 59583a7d7ae042219fabd31cf4a8314a
I20260927 10:53:35.700747 20016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:35.711297 20016 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:35.712206 20016 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:35.721093 20016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bba6ece1b4b74bfea3c55d7c6f6799c0: Generated new TSK 0
I20260927 10:53:35.721762 20016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:53:35.734992 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:35.729846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51104 (local address 127.14.68.62:45617)
0927 10:53:35.729982 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:35.729988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:35.730402 (+ 414us) server_negotiation.cc:407] Connection header received
0927 10:53:35.730548 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:35.730550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:35.730595 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:35.730679 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0927 10:53:35.730943 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:35.731464 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:35.732740 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:35.732950 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:35.733025 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:35.733336 (+ 311us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 10:53:35.734648 (+ 1312us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37083/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 10:53:35.734866 (+ 218us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51104: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37083/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 10:53:35.735052 20005 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51104: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37083/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 10:53:35.736960 20031 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:35.729847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:45617 (local address 127.0.0.1:51104)
0927 10:53:35.730296 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:35.730311 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:35.730433 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:35.730707 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:35.730710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:35.730718 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 10:53:35.730841 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:35.730847 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:35.731572 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:35.731576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:35.732589 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:35.732600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:35.733060 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:35.733064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:35.733152 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:35.733179 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 10:53:35.734811 (+ 1632us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37083/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 10:53:35.736786 (+ 1975us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.68.62:45617: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37083/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":1786,"thread_start_us":133,"threads_started":1}
I20260927 10:53:35.737495 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 19935
E20260927 10:53:35.746133 20032 webserver.cc:533] Webserver: sslize failed:
E20260927 10:53:35.746192 20032 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T10:53:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (291 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T10:53:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:35Z Disabled control of system clock
I20260927 10:53:35.783633 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:33581
--webserver_interface=127.14.68.62
--webserver_port=0
--builtin_ntp_servers=127.14.68.20:36849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:33581
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506389460539-14608-0/testchainca.cert with env {}
W20260927 10:53:35.892640 20036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:35.892925 20036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:35.892961 20036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:35.896981 20036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:35.897032 20036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:35.897047 20036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:35.897064 20036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:35.897135 20036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 10:53:35.897163 20036 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 10:53:35.897197 20036 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 10:53:35.902468 20036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:36849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:33581
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:33581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506389460539-14608-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:35.903481 20036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:35.904517 20036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:35.912985 20041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:35.913038 20036 server_base.cc:1034] running on GCE node
W20260927 10:53:35.913025 20044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:35.912985 20042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:35.913827 20036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:35.914916 20036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:35.916072 20036 hybrid_clock.cc:648] HybridClock initialized: now 1790506415916045 us; error 38 us; skew 500 ppm
I20260927 10:53:35.917958 20036 webserver.cc:460] Webserver started at http://127.14.68.62:44389/ using document root <none> and password file <none>
I20260927 10:53:35.918517 20036 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:35.918579 20036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:35.918830 20036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:35.920845 20036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "1a136d2ad2774890b14f994289a82c9c"
format_stamp: "Formatted at 2026-09-27 10:53:35 on dist-test-slave-tz51"
server_key: "fd63ed4860583e680757c5066f7902cd"
server_key_iv: "e4a254b4cfbcba6e1d9a09698371825d"
server_key_version: "encryptionkey@0"
I20260927 10:53:35.921407 20036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "1a136d2ad2774890b14f994289a82c9c"
format_stamp: "Formatted at 2026-09-27 10:53:35 on dist-test-slave-tz51"
server_key: "fd63ed4860583e680757c5066f7902cd"
server_key_iv: "e4a254b4cfbcba6e1d9a09698371825d"
server_key_version: "encryptionkey@0"
I20260927 10:53:35.925066 20036 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:53:35.927404 20050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:35.928673 20036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:35.928808 20036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "1a136d2ad2774890b14f994289a82c9c"
format_stamp: "Formatted at 2026-09-27 10:53:35 on dist-test-slave-tz51"
server_key: "fd63ed4860583e680757c5066f7902cd"
server_key_iv: "e4a254b4cfbcba6e1d9a09698371825d"
server_key_version: "encryptionkey@0"
I20260927 10:53:35.928921 20036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:35.946844 20036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:35.947536 20036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:35.947790 20036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:35.954563 20036 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:33581
I20260927 10:53:35.954573 20102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:33581 every 8 connection(s)
I20260927 10:53:35.955466 20036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:35.958272 20103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:35.959033 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 20036
I20260927 10:53:35.959167 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:53:35.959494 14608 external_mini_cluster.cc:1428] Setting key d749c7624a7214422d7def2c455328e7
I20260927 10:53:35.964195 20103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a136d2ad2774890b14f994289a82c9c: Bootstrap starting.
I20260927 10:53:35.966338 20103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a136d2ad2774890b14f994289a82c9c: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:35.967145 20103 log.cc:824] T 00000000000000000000000000000000 P 1a136d2ad2774890b14f994289a82c9c: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:35.968597 20106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:35.960904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.14.68.62:33581)
0927 10:53:35.961524 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:35.961538 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:35.961571 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:53:35.962185 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:35.962201 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:35.962590 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:35.962920 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:35.963333 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:35.964391 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:35.965286 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:35.965745 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:35.965915 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:35.966795 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:35.966800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:35.966811 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:53:35.967206 (+ 395us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:35.967285 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:53:35.967330 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:35.967404 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:35.967598 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 10:53:35.967875 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":141,"threads_started":1}
I20260927 10:53:35.969091 20103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a136d2ad2774890b14f994289a82c9c: No bootstrap required, opened a new log
I20260927 10:53:35.971393 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:53:35.973332 20103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a136d2ad2774890b14f994289a82c9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a136d2ad2774890b14f994289a82c9c" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33581 } }
I20260927 10:53:35.973541 20103 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a136d2ad2774890b14f994289a82c9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:35.973573 20103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a136d2ad2774890b14f994289a82c9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a136d2ad2774890b14f994289a82c9c, State: Initialized, Role: FOLLOWER
I20260927 10:53:35.973896 20103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a136d2ad2774890b14f994289a82c9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a136d2ad2774890b14f994289a82c9c" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33581 } }
I20260927 10:53:35.973999 20103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a136d2ad2774890b14f994289a82c9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:35.974061 20103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a136d2ad2774890b14f994289a82c9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:35.974128 20103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a136d2ad2774890b14f994289a82c9c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:35.974918 20103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a136d2ad2774890b14f994289a82c9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a136d2ad2774890b14f994289a82c9c" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33581 } }
I20260927 10:53:35.975188 20103 leader_election.cc:302] T 00000000000000000000000000000000 P 1a136d2ad2774890b14f994289a82c9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a136d2ad2774890b14f994289a82c9c; no voters:
I20260927 10:53:35.975417 20103 leader_election.cc:288] T 00000000000000000000000000000000 P 1a136d2ad2774890b14f994289a82c9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:35.975557 20108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a136d2ad2774890b14f994289a82c9c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:35.975821 20108 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a136d2ad2774890b14f994289a82c9c [term 1 LEADER]: Becoming Leader. State: Replica: 1a136d2ad2774890b14f994289a82c9c, State: Running, Role: LEADER
I20260927 10:53:35.976125 20108 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a136d2ad2774890b14f994289a82c9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a136d2ad2774890b14f994289a82c9c" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33581 } }
I20260927 10:53:35.976609 20103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a136d2ad2774890b14f994289a82c9c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:35.977643 20109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a136d2ad2774890b14f994289a82c9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a136d2ad2774890b14f994289a82c9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a136d2ad2774890b14f994289a82c9c" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33581 } } }
I20260927 10:53:35.977771 20109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a136d2ad2774890b14f994289a82c9c [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:35.978168 20117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:35.978168 20110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a136d2ad2774890b14f994289a82c9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a136d2ad2774890b14f994289a82c9c. Latest consensus state: current_term: 1 leader_uuid: "1a136d2ad2774890b14f994289a82c9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a136d2ad2774890b14f994289a82c9c" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33581 } } }
I20260927 10:53:35.978379 20110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a136d2ad2774890b14f994289a82c9c [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:35.980772 20117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:35.986261 20117 catalog_manager.cc:1347] Generated new cluster ID: cb4703ee21dc44dcab12fd66b0882cef
I20260927 10:53:35.986330 20117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:36.018374 20117 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:36.019349 20117 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:36.027956 20117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a136d2ad2774890b14f994289a82c9c: Generated new TSK 0
I20260927 10:53:36.028780 20117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:53:36.096063 20106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:36.036619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51780 (local address 127.14.68.62:33581)
0927 10:53:36.036745 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:36.036750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:36.036932 (+ 182us) server_negotiation.cc:407] Connection header received
0927 10:53:36.037077 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:36.037079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:36.037141 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:36.037227 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0927 10:53:36.037511 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:36.038004 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:36.039267 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:36.039490 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:36.039565 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:36.039929 (+ 364us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 10:53:36.095588 (+ 55659us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 10:53:36.095944 (+ 356us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51780: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 10:53:36.096167 20106 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51780: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 10:53:36.096621 20132 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:36.036542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:33581 (local address 127.0.0.1:51780)
0927 10:53:36.036840 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:36.036856 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:36.036968 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:36.037262 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:36.037266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:36.037274 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 10:53:36.037410 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:36.037416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:36.038129 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:36.038135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:36.039153 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:36.039163 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:36.039614 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:36.039618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:36.039757 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:36.039784 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 10:53:36.095883 (+ 56099us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 10:53:36.096104 (+ 221us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.68.62:33581: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":105,"threads_started":1}
I20260927 10:53:36.097343 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 20036
2026-09-27T10:53:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (340 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (117 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (119 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:53:36 dist-test-slave-tz51 krb5kdc[20144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:53:36 dist-test-slave-tz51 krb5kdc[20144](info): set up 2 sockets
Sep 27 10:53:36 dist-test-slave-tz51 krb5kdc[20144](info): commencing operation
krb5kdc: starting...
W20260927 10:53:37.364032 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:53:37 dist-test-slave-tz51 krb5kdc[20144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:53:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:53:37.480850 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:36109
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:39509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:36109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:37.588208 20157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:37.588469 20157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:37.588510 20157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:37.592540 20157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:37.592595 20157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:37.592614 20157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:37.592640 20157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:37.592656 20157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:37.597975 20157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:39509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:36109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:36109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:37.599007 20157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:37.600032 20157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:37.608307 20163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:37.608279 20162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:37.608290 20165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:37.608705 20157 server_base.cc:1034] running on GCE node
I20260927 10:53:37.609290 20157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:37.610461 20157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:37.611650 20157 hybrid_clock.cc:648] HybridClock initialized: now 1790506417611635 us; error 34 us; skew 500 ppm
Sep 27 10:53:37 dist-test-slave-tz51 krb5kdc[20144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506417, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:37.619197 20157 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:53:37.620133 20157 webserver.cc:460] Webserver started at http://127.14.68.62:33739/ using document root <none> and password file <none>
I20260927 10:53:37.620695 20157 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:37.620795 20157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:37.621022 20157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:37.623006 20157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "b4cb238ab8084343ac474f917ff3fa3d"
format_stamp: "Formatted at 2026-09-27 10:53:37 on dist-test-slave-tz51"
server_key: "b65014c61b4349dc9ccb37446c5e8b82"
server_key_iv: "a08d0688d31dfcdc39987cc63f02709c"
server_key_version: "encryptionkey@0"
I20260927 10:53:37.623535 20157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "b4cb238ab8084343ac474f917ff3fa3d"
format_stamp: "Formatted at 2026-09-27 10:53:37 on dist-test-slave-tz51"
server_key: "b65014c61b4349dc9ccb37446c5e8b82"
server_key_iv: "a08d0688d31dfcdc39987cc63f02709c"
server_key_version: "encryptionkey@0"
I20260927 10:53:37.627084 20157 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:53:37.629415 20172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:37.630507 20157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 10:53:37.630645 20157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "b4cb238ab8084343ac474f917ff3fa3d"
format_stamp: "Formatted at 2026-09-27 10:53:37 on dist-test-slave-tz51"
server_key: "b65014c61b4349dc9ccb37446c5e8b82"
server_key_iv: "a08d0688d31dfcdc39987cc63f02709c"
server_key_version: "encryptionkey@0"
I20260927 10:53:37.630751 20157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:37.648430 20157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:37.651362 20157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:37.651575 20157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:37.659252 20157 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:36109
I20260927 10:53:37.659281 20224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:36109 every 8 connection(s)
I20260927 10:53:37.660272 20157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:37.662992 20225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:37.666450 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 20157
I20260927 10:53:37.666554 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:53:37.666852 14608 external_mini_cluster.cc:1428] Setting key 9c7a3eec316963f6b6e11d6e4674a1a8
I20260927 10:53:37.668720 20225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d: Bootstrap starting.
I20260927 10:53:37.670894 20225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:37.671595 20225 log.cc:824] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:37.673662 20225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d: No bootstrap required, opened a new log
Sep 27 10:53:37 dist-test-slave-tz51 krb5kdc[20144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:37.677551 20225 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4cb238ab8084343ac474f917ff3fa3d" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 36109 } }
I20260927 10:53:37.677798 20225 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:37.677877 20225 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4cb238ab8084343ac474f917ff3fa3d, State: Initialized, Role: FOLLOWER
I20260927 10:53:37.678352 20225 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4cb238ab8084343ac474f917ff3fa3d" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 36109 } }
I20260927 10:53:37.678524 20225 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:37.678594 20225 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:37.678704 20225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:37.679728 20225 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4cb238ab8084343ac474f917ff3fa3d" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 36109 } }
I20260927 10:53:37.680080 20225 leader_election.cc:302] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4cb238ab8084343ac474f917ff3fa3d; no voters:
I20260927 10:53:37.680465 20225 leader_election.cc:288] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:37.680563 20230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:37.680763 20230 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d [term 1 LEADER]: Becoming Leader. State: Replica: b4cb238ab8084343ac474f917ff3fa3d, State: Running, Role: LEADER
I20260927 10:53:37.681207 20230 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4cb238ab8084343ac474f917ff3fa3d" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 36109 } }
I20260927 10:53:37.681577 20225 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:37.681594 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:37.668263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47192 (local address 127.14.68.62:36109)
0927 10:53:37.668686 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:37.668700 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:37.668741 (+ 41us) server_negotiation.cc:407] Connection header received
0927 10:53:37.669381 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:37.669399 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:37.669755 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:37.670082 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:37.670929 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:37.672068 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:37.673239 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:37.673682 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:37.673849 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:37.676489 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:37.676494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:37.676505 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:37.676530 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:37.678277 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:37.678766 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:37.678771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:37.678775 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:37.678921 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:37.679213 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:37.679215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:37.679216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:37.679568 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:37.679797 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:37.680029 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 10:53:37.680387 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":129,"threads_started":1}
I20260927 10:53:37.683224 20231 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4cb238ab8084343ac474f917ff3fa3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4cb238ab8084343ac474f917ff3fa3d" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 36109 } } }
I20260927 10:53:37.683269 20232 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4cb238ab8084343ac474f917ff3fa3d. Latest consensus state: current_term: 1 leader_uuid: "b4cb238ab8084343ac474f917ff3fa3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4cb238ab8084343ac474f917ff3fa3d" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 36109 } } }
I20260927 10:53:37.683359 20232 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:37.683358 20231 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d [sys.catalog]: This master's current role is: LEADER
W20260927 10:53:37.686486 20245 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:53:37.686564 20245 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:53:37.686666 20243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:37.687443 20243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:37.693326 20243 catalog_manager.cc:1347] Generated new cluster ID: 7d71ae3cdf624d579f9845030d1c2e11
I20260927 10:53:37.693387 20243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:37.716475 20243 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:37.717759 20243 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:37.729907 20243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4cb238ab8084343ac474f917ff3fa3d: Generated new TSK 0
I20260927 10:53:37.730759 20243 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
I20260927 10:53:37.758317 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:36109
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:39509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:37.869428 20251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:37.869678 20251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:37.869720 20251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:37.873908 20251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:37.873958 20251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:37.874089 20251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:53:37.879235 20251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:39509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:36109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:37.880263 20251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:37.881317 20251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:37.891135 20259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:37.891132 20256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:37.891146 20257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:37.891477 20251 server_base.cc:1034] running on GCE node
I20260927 10:53:37.891911 20251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:37.892532 20251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:37.893688 20251 hybrid_clock.cc:648] HybridClock initialized: now 1790506417893670 us; error 31 us; skew 500 ppm
Sep 27 10:53:37 dist-test-slave-tz51 krb5kdc[20144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506417, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:37.896564 20251 init.cc:377] Logged in from keytab as kudu/127.14.68.1@KRBTEST.COM (short username kudu)
I20260927 10:53:37.897399 20251 webserver.cc:460] Webserver started at http://127.14.68.1:46205/ using document root <none> and password file <none>
I20260927 10:53:37.897938 20251 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:37.897997 20251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:37.898247 20251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:37.900308 20251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "d43737489aea46228c0d88f11512e11a"
format_stamp: "Formatted at 2026-09-27 10:53:37 on dist-test-slave-tz51"
server_key: "68c1c579230c23f39dbb80049e9d3f0b"
server_key_iv: "ecc7bf94df75ac090a3933e625299b9d"
server_key_version: "encryptionkey@0"
I20260927 10:53:37.900854 20251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "d43737489aea46228c0d88f11512e11a"
format_stamp: "Formatted at 2026-09-27 10:53:37 on dist-test-slave-tz51"
server_key: "68c1c579230c23f39dbb80049e9d3f0b"
server_key_iv: "ecc7bf94df75ac090a3933e625299b9d"
server_key_version: "encryptionkey@0"
I20260927 10:53:37.904536 20251 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:53:37.906906 20266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:37.908160 20251 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:37.908274 20251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "d43737489aea46228c0d88f11512e11a"
format_stamp: "Formatted at 2026-09-27 10:53:37 on dist-test-slave-tz51"
server_key: "68c1c579230c23f39dbb80049e9d3f0b"
server_key_iv: "ecc7bf94df75ac090a3933e625299b9d"
server_key_version: "encryptionkey@0"
I20260927 10:53:37.908375 20251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:37.922391 20251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:37.925554 20251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:37.925741 20251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:37.926311 20251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:37.927079 20251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:37.927137 20251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:37.927215 20251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:37.927248 20251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:37.936450 20251 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:40821
I20260927 10:53:37.936470 20379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:40821 every 8 connection(s)
I20260927 10:53:37.937397 20251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:53:37.944360 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 20251
I20260927 10:53:37.944469 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:53:37.944785 14608 external_mini_cluster.cc:1428] Setting key 42ebef53092609d9b791aa2eb4b71521
Sep 27 10:53:37 dist-test-slave-tz51 krb5kdc[20144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506417, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:37.952011 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:37.939321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:59841 (local address 127.14.68.62:36109)
0927 10:53:37.939464 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:37.939468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:37.940330 (+ 862us) server_negotiation.cc:407] Connection header received
0927 10:53:37.941693 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:37.941697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:37.941754 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:37.941853 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:37.943013 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:37.943566 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:37.944843 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:37.945118 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:37.945247 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:37.948307 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:37.948311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:37.948314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:37.948344 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:37.949721 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:37.950362 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:37.950364 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:37.950366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:37.950435 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:37.950863 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:37.950867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:37.950868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:37.951050 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:37.951159 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:37.951664 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 10:53:37.951834 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:53:37.952860 20382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:37.939665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:36109 (local address 127.14.68.1:59841)
0927 10:53:37.940181 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:37.940222 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:37.941460 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:37.942027 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:37.942034 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:37.942385 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:37.942862 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:37.942876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:37.943728 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:37.943735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:37.944720 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:37.944730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:37.945298 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:37.945301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:37.945529 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:37.946214 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:37.946232 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:37.948047 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:37.949840 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:37.949849 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:37.949868 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:37.950256 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:37.950530 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:37.950534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:37.950537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:37.950682 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:37.951159 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:37.951167 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:37.951422 (+ 255us) client_negotiation.cc:769] Sending connection context
0927 10:53:37.951610 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 10:53:37.951965 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":115,"threads_started":1}
I20260927 10:53:37.954139 20380 heartbeater.cc:347] Connected to a master server at 127.14.68.62:36109
I20260927 10:53:37.954407 20380 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.14.68.2@KRBTEST.COM; defaulting to no policy
I20260927 10:53:37.954936 20380 heartbeater.cc:511] Master 127.14.68.62:36109 requested a full tablet report, sending...
I20260927 10:53:37.956569 20187 ts_manager.cc:194] Registered new tserver with Master: d43737489aea46228c0d88f11512e11a (127.14.68.1:40821)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.2@KRBTEST.COM" created.
I20260927 10:53:37.957769 20187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:59841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
I20260927 10:53:37.972209 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:36109
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:39509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:38.082962 20385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:38.083199 20385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:38.083271 20385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:38.087438 20385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:38.087497 20385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:38.087553 20385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:53:38.092758 20385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:39509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:36109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:38.093786 20385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:38.094770 20385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:38.103972 20390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:38.103972 20391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:38.104091 20385 server_base.cc:1034] running on GCE node
W20260927 10:53:38.104005 20393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:38.104708 20385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:38.105307 20385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:38.106499 20385 hybrid_clock.cc:648] HybridClock initialized: now 1790506418106484 us; error 26 us; skew 500 ppm
Sep 27 10:53:38 dist-test-slave-tz51 krb5kdc[20144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506418, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:38.109259 20385 init.cc:377] Logged in from keytab as kudu/127.14.68.2@KRBTEST.COM (short username kudu)
I20260927 10:53:38.110067 20385 webserver.cc:460] Webserver started at http://127.14.68.2:40497/ using document root <none> and password file <none>
I20260927 10:53:38.110556 20385 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:38.110621 20385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:38.110776 20385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:38.112848 20385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "2a69e5e832484961ae7282f495919393"
format_stamp: "Formatted at 2026-09-27 10:53:38 on dist-test-slave-tz51"
server_key: "7e42583bda82e13c81c09af074479ba0"
server_key_iv: "d6f7a54c5dd84b143f9c2200941a6683"
server_key_version: "encryptionkey@0"
I20260927 10:53:38.113425 20385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "2a69e5e832484961ae7282f495919393"
format_stamp: "Formatted at 2026-09-27 10:53:38 on dist-test-slave-tz51"
server_key: "7e42583bda82e13c81c09af074479ba0"
server_key_iv: "d6f7a54c5dd84b143f9c2200941a6683"
server_key_version: "encryptionkey@0"
I20260927 10:53:38.117051 20385 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:53:38.119421 20400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:38.120780 20385 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:38.120901 20385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "2a69e5e832484961ae7282f495919393"
format_stamp: "Formatted at 2026-09-27 10:53:38 on dist-test-slave-tz51"
server_key: "7e42583bda82e13c81c09af074479ba0"
server_key_iv: "d6f7a54c5dd84b143f9c2200941a6683"
server_key_version: "encryptionkey@0"
I20260927 10:53:38.121004 20385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:38.146224 20385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:38.150027 20385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:38.150352 20385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:38.150995 20385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:38.152040 20385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:38.152091 20385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:38.152253 20385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:38.152318 20385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:38.161042 20385 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:41051
I20260927 10:53:38.161100 20513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:41051 every 8 connection(s)
I20260927 10:53:38.161952 20385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:53:38.168210 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 20385
I20260927 10:53:38.168360 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:53:38.168644 14608 external_mini_cluster.cc:1428] Setting key 54687211f0a8cb16abeab0da5e6db18a
Sep 27 10:53:38 dist-test-slave-tz51 krb5kdc[20144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506418, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:38.176340 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:38.163769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:57649 (local address 127.14.68.62:36109)
0927 10:53:38.163896 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:38.163900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:38.164770 (+ 870us) server_negotiation.cc:407] Connection header received
0927 10:53:38.166165 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:38.166168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:38.166220 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:38.166327 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:38.167443 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:38.168036 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:38.169447 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:38.169670 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:38.169780 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:38.172330 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:38.172335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:38.172340 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:38.172371 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:38.174068 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:38.174665 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:38.174670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:38.174672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:38.174780 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:38.175111 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:38.175118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:38.175122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:38.175378 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:38.175495 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:38.175998 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 10:53:38.176136 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:53:38.177042 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:38.164163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:36109 (local address 127.14.68.2:57649)
0927 10:53:38.164625 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:38.164669 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:38.165918 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:38.166493 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:38.166499 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:38.166840 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:38.167304 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:38.167317 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:38.168193 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:38.168197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:38.169318 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:38.169327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:38.169813 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:38.169817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:38.169996 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:38.170566 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:38.170581 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:38.172165 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:38.174224 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:38.174259 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:38.174274 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:38.174548 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:38.174885 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:38.174887 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:38.174889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:38.175013 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:38.175486 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:38.175491 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:38.175755 (+ 264us) client_negotiation.cc:769] Sending connection context
0927 10:53:38.175915 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 10:53:38.176147 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":117,"threads_started":1}
I20260927 10:53:38.178253 20514 heartbeater.cc:347] Connected to a master server at 127.14.68.62:36109
WARNING: no policy specified for kudu/127.14.68.3@KRBTEST.COM; defaulting to no policy
I20260927 10:53:38.178493 20514 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:38.178946 20514 heartbeater.cc:511] Master 127.14.68.62:36109 requested a full tablet report, sending...
I20260927 10:53:38.180008 20187 ts_manager.cc:194] Registered new tserver with Master: 2a69e5e832484961ae7282f495919393 (127.14.68.2:41051)
I20260927 10:53:38.180580 20187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:57649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
I20260927 10:53:38.197163 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:36109
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:39509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:38.307925 20519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:38.308199 20519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:38.308290 20519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:38.312404 20519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:38.312477 20519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:38.312563 20519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:53:38.317804 20519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:39509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:36109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:38.318849 20519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:38.319939 20519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:38.329306 20524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:38.329337 20527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:38.329341 20525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:38.329712 20519 server_base.cc:1034] running on GCE node
I20260927 10:53:38.330118 20519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:38.330713 20519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:38.331874 20519 hybrid_clock.cc:648] HybridClock initialized: now 1790506418331849 us; error 30 us; skew 500 ppm
Sep 27 10:53:38 dist-test-slave-tz51 krb5kdc[20144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506418, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:38.334594 20519 init.cc:377] Logged in from keytab as kudu/127.14.68.3@KRBTEST.COM (short username kudu)
I20260927 10:53:38.335451 20519 webserver.cc:460] Webserver started at http://127.14.68.3:35945/ using document root <none> and password file <none>
I20260927 10:53:38.336067 20519 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:38.336156 20519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:38.336412 20519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:38.338408 20519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "5a5ffae13a3248e59a8ebef6ebac6e77"
format_stamp: "Formatted at 2026-09-27 10:53:38 on dist-test-slave-tz51"
server_key: "024481b99244596cbf3873d99bbdaf3f"
server_key_iv: "5cd0b71d716f92ee39ccca2a7a4db54e"
server_key_version: "encryptionkey@0"
I20260927 10:53:38.338982 20519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "5a5ffae13a3248e59a8ebef6ebac6e77"
format_stamp: "Formatted at 2026-09-27 10:53:38 on dist-test-slave-tz51"
server_key: "024481b99244596cbf3873d99bbdaf3f"
server_key_iv: "5cd0b71d716f92ee39ccca2a7a4db54e"
server_key_version: "encryptionkey@0"
I20260927 10:53:38.342654 20519 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:53:38.345213 20534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:38.346449 20519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:53:38.346586 20519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "5a5ffae13a3248e59a8ebef6ebac6e77"
format_stamp: "Formatted at 2026-09-27 10:53:38 on dist-test-slave-tz51"
server_key: "024481b99244596cbf3873d99bbdaf3f"
server_key_iv: "5cd0b71d716f92ee39ccca2a7a4db54e"
server_key_version: "encryptionkey@0"
I20260927 10:53:38.346706 20519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:38.370618 20519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:38.373441 20519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:38.373641 20519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:38.374243 20519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:38.375056 20519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:38.375125 20519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:38.375204 20519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:38.375257 20519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:38.384657 20519 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:39457
I20260927 10:53:38.384680 20647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:39457 every 8 connection(s)
I20260927 10:53:38.385548 20519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:53:38.393375 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 20519
I20260927 10:53:38.393509 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:53:38.393904 14608 external_mini_cluster.cc:1428] Setting key 286eab93b86e7346951259f3b1978515
Sep 27 10:53:38 dist-test-slave-tz51 krb5kdc[20144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506418, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:38.399605 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:38.387395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:51141 (local address 127.14.68.62:36109)
0927 10:53:38.387529 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:38.387533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:38.388427 (+ 894us) server_negotiation.cc:407] Connection header received
0927 10:53:38.389780 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:38.389783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:38.389839 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:38.389942 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:38.391082 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:38.391616 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:38.392797 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:38.393024 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:38.393160 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:38.396060 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:38.396064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:38.396067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:38.396101 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:38.397590 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:38.398113 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:38.398117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:38.398118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:38.398202 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:38.398553 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:38.398556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:38.398558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:38.398740 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:38.398843 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:38.399328 (+ 485us) server_negotiation.cc:299] Negotiation successful
0927 10:53:38.399444 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:53:38.400420 20650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:38.387827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:36109 (local address 127.14.68.3:51141)
0927 10:53:38.388275 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:38.388317 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:38.389559 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:38.390117 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:38.390125 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:38.390475 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:38.390939 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:38.390952 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:38.391799 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:38.391808 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:38.392669 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:38.392678 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:38.393185 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:38.393190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:38.393367 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:38.394090 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:38.394106 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:38.395902 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:38.397723 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:38.397729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:38.397745 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:38.398002 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:38.398302 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:38.398305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:38.398307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:38.398451 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:38.398841 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:38.398848 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:38.399094 (+ 246us) client_negotiation.cc:769] Sending connection context
0927 10:53:38.399278 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 10:53:38.399603 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":110,"threads_started":1}
I20260927 10:53:38.401489 20648 heartbeater.cc:347] Connected to a master server at 127.14.68.62:36109
I20260927 10:53:38.401701 20648 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:38.402220 20648 heartbeater.cc:511] Master 127.14.68.62:36109 requested a full tablet report, sending...
I20260927 10:53:38.403401 20187 ts_manager.cc:194] Registered new tserver with Master: 5a5ffae13a3248e59a8ebef6ebac6e77 (127.14.68.3:39457)
I20260927 10:53:38.404042 20187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:51141
I20260927 10:53:38.408012 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/test-admin.keytab.
Sep 27 10:53:38 dist-test-slave-tz51 krb5kdc[20144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:38.425273 14608 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 10:53:38.427981 20655 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:38.426557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38130 (local address 127.14.68.1:40821)
0927 10:53:38.426876 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:38.426890 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:38.426914 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:53:38.426963 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:38.426968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:38.427148 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:38.427274 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:38.427778 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38130: BlockingRecv error: recv got EOF from 127.0.0.1:38130 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":126,"threads_started":1}
W20260927 10:53:38.428054 14608 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/krb5cc' of type FILE
Sep 27 10:53:38 dist-test-slave-tz51 krb5kdc[20144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:38.429042 14608 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 10:53:38 dist-test-slave-tz51 krb5kdc[20144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:38.429916 14608 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 10:53:38 dist-test-slave-tz51 krb5kdc[20144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.1@KRBTEST.COM
I20260927 10:53:38.439360 20655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:38.430678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38144 (local address 127.14.68.1:40821)
0927 10:53:38.430807 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:38.430810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:38.430854 (+ 44us) server_negotiation.cc:407] Connection header received
0927 10:53:38.430941 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:38.430943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:38.430995 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:38.431066 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:38.431602 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:38.432433 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:38.433220 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:38.433663 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:38.433762 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:38.436134 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:38.436143 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:38.436156 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:38.436178 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:38.437650 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:38.438075 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:38.438080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:38.438081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:38.438155 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:38.438458 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:38.438460 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:38.438461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:38.438697 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:38.438853 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:38.439033 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 10:53:38.439205 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 10:53:38.443318 20655 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:38.442287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.14.68.1:40821)
0927 10:53:38.442414 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:38.442418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:38.442517 (+ 99us) server_negotiation.cc:407] Connection header received
0927 10:53:38.442654 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:38.442657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:38.442700 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:38.442789 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:38.443191 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38150: BlockingRecv error: recv got EOF from 127.0.0.1:38150 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 10:53:38.443466 14608 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/krb5cc' of type FILE
Sep 27 10:53:38 dist-test-slave-tz51 krb5kdc[20144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:38.444634 14608 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 10:53:38 dist-test-slave-tz51 krb5kdc[20144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:38.445487 14608 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 10:53:38 dist-test-slave-tz51 krb5kdc[20144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.1@KRBTEST.COM
I20260927 10:53:38.453971 20655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:38.446322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.14.68.1:40821)
0927 10:53:38.446443 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:38.446447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:38.446552 (+ 105us) server_negotiation.cc:407] Connection header received
0927 10:53:38.446698 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:38.446701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:38.446743 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:38.446824 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:38.447314 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:38.447898 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:38.448674 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:38.448961 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:38.449071 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:38.451120 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:38.451123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:38.451125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:38.451145 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:38.452516 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:38.452885 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:38.452887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:38.452888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:38.452955 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:38.453229 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:38.453231 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:38.453233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:38.453376 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:38.453467 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:38.453692 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 10:53:38.453810 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 10:53:38.456739 20655 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:38.455793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38182 (local address 127.14.68.1:40821)
0927 10:53:38.455905 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:38.455909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:38.455953 (+ 44us) server_negotiation.cc:407] Connection header received
0927 10:53:38.456090 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:38.456093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:38.456132 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:38.456233 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:38.456647 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38182: BlockingRecv error: recv got EOF from 127.0.0.1:38182 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 10:53:38.456871 14608 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506389460539-14608-0/krb5kdc/krb5cc' of type FILE
Sep 27 10:53:38 dist-test-slave-tz51 krb5kdc[20144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:38.457800 14608 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 10:53:38 dist-test-slave-tz51 krb5kdc[20144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:38.458689 14608 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 10:53:38 dist-test-slave-tz51 krb5kdc[20144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.1@KRBTEST.COM
I20260927 10:53:38.466554 20655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:38.459432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.14.68.1:40821)
0927 10:53:38.459556 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:38.459558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:38.459591 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:53:38.459732 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:38.459735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:38.459774 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:38.459842 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:38.460325 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:38.460794 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:38.461529 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:38.461740 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:38.461811 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:38.464034 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:38.464037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:38.464039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:38.464059 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:38.465199 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:38.465570 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:38.465572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:38.465573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:38.465638 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:38.465850 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:38.465852 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:38.465853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:38.466000 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:38.466103 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:38.466281 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 10:53:38.466387 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:53:38.467554 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 20251
I20260927 10:53:38.473238 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 20385
I20260927 10:53:38.478398 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 20519
I20260927 10:53:38.483664 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 20157
2026-09-27T10:53:38Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2151 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:53:38 dist-test-slave-tz51 krb5kdc[20668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:53:38 dist-test-slave-tz51 krb5kdc[20668](info): set up 2 sockets
Sep 27 10:53:38 dist-test-slave-tz51 krb5kdc[20668](info): commencing operation
krb5kdc: starting...
W20260927 10:53:39.521487 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:53:39 dist-test-slave-tz51 krb5kdc[20668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:53:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:39Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.62.keytab.
Entry for principal oryx/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.62.keytab.
I20260927 10:53:39.644868 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:40605
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.62.keytab
--principal=oryx/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:46363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:40605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:39.757584 20681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:39.757864 20681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:39.757936 20681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:39.762001 20681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:39.762105 20681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:39.762143 20681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:39.762200 20681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:39.762238 20681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:39.767448 20681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:46363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:40605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.62.keytab
--principal=oryx/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:40605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:39.768442 20681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:39.769418 20681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:39.777803 20686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:39.777827 20687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:39.777937 20681 server_base.cc:1034] running on GCE node
W20260927 10:53:39.777810 20689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:39.778632 20681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:39.779670 20681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:39.780886 20681 hybrid_clock.cc:648] HybridClock initialized: now 1790506419780870 us; error 29 us; skew 500 ppm
Sep 27 10:53:39 dist-test-slave-tz51 krb5kdc[20668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506419, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:39.783540 20681 init.cc:377] Logged in from keytab as oryx/127.14.68.62@KRBTEST.COM (short username oryx)
I20260927 10:53:39.784387 20681 webserver.cc:460] Webserver started at http://127.14.68.62:39407/ using document root <none> and password file <none>
I20260927 10:53:39.784885 20681 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:39.784943 20681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:39.785113 20681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:39.787016 20681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "e37e387fd038459890cc0553983720ae"
format_stamp: "Formatted at 2026-09-27 10:53:39 on dist-test-slave-tz51"
server_key: "51ac21b91c8edf85033abd90c2e5e363"
server_key_iv: "ed3d7382f03ff590bd719d078cfb5315"
server_key_version: "encryptionkey@0"
I20260927 10:53:39.787508 20681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "e37e387fd038459890cc0553983720ae"
format_stamp: "Formatted at 2026-09-27 10:53:39 on dist-test-slave-tz51"
server_key: "51ac21b91c8edf85033abd90c2e5e363"
server_key_iv: "ed3d7382f03ff590bd719d078cfb5315"
server_key_version: "encryptionkey@0"
I20260927 10:53:39.791446 20681 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 10:53:39.793938 20696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:39.795210 20681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:53:39.795315 20681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "e37e387fd038459890cc0553983720ae"
format_stamp: "Formatted at 2026-09-27 10:53:39 on dist-test-slave-tz51"
server_key: "51ac21b91c8edf85033abd90c2e5e363"
server_key_iv: "ed3d7382f03ff590bd719d078cfb5315"
server_key_version: "encryptionkey@0"
I20260927 10:53:39.795403 20681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:39.822814 20681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:39.826591 20681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:39.826807 20681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:39.834556 20681 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:40605
I20260927 10:53:39.834865 20748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:40605 every 8 connection(s)
I20260927 10:53:39.835598 20681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:39.838424 20749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:39.840938 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 20681
I20260927 10:53:39.841130 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:53:39.841426 14608 external_mini_cluster.cc:1428] Setting key 7b860b9336a4f5af291097bae8cfc949
I20260927 10:53:39.844576 20749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae: Bootstrap starting.
I20260927 10:53:39.847258 20749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:39.848348 20749 log.cc:824] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:39.850493 20749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae: No bootstrap required, opened a new log
Sep 27 10:53:39 dist-test-slave-tz51 krb5kdc[20668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM
I20260927 10:53:39.853710 20749 raft_consensus.cc:348] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } }
I20260927 10:53:39.853977 20749 raft_consensus.cc:374] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:39.854058 20749 raft_consensus.cc:729] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e37e387fd038459890cc0553983720ae, State: Initialized, Role: FOLLOWER
I20260927 10:53:39.854598 20749 consensus_queue.cc:260] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } }
I20260927 10:53:39.854791 20749 raft_consensus.cc:388] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:39.854871 20749 raft_consensus.cc:482] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:39.854990 20749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:39.856209 20749 raft_consensus.cc:504] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } }
I20260927 10:53:39.856581 20749 leader_election.cc:302] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e37e387fd038459890cc0553983720ae; no voters:
I20260927 10:53:39.856938 20749 leader_election.cc:288] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:39.857039 20754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:39.857252 20754 raft_consensus.cc:686] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [term 1 LEADER]: Becoming Leader. State: Replica: e37e387fd038459890cc0553983720ae, State: Running, Role: LEADER
I20260927 10:53:39.857372 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:39.842809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40746 (local address 127.14.68.62:40605)
0927 10:53:39.843218 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:39.843234 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:39.843270 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:53:39.843980 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:39.844000 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:39.844351 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:39.844652 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:39.845601 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:39.846630 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:39.847891 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:39.848379 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:39.848566 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:39.851496 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:39.851502 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:39.851517 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:39.851548 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:39.853965 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:39.854491 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:39.854496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:39.854500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:39.854636 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:39.854993 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:39.854996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:39.854997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:39.855464 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:39.855628 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:39.855970 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 10:53:39.856300 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":137,"threads_started":1}
I20260927 10:53:39.857676 20754 consensus_queue.cc:237] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } }
I20260927 10:53:39.858354 20749 sys_catalog.cc:564] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:39.859437 20756 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader e37e387fd038459890cc0553983720ae. Latest consensus state: current_term: 1 leader_uuid: "e37e387fd038459890cc0553983720ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } }
I20260927 10:53:39.860134 20756 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:39.860994 20755 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e37e387fd038459890cc0553983720ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } }
I20260927 10:53:39.861092 20755 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [sys.catalog]: This master's current role is: LEADER
W20260927 10:53:39.863380 20769 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:53:39.863456 20769 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:53:39.863626 20768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:39.864387 20768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:39.869938 20768 catalog_manager.cc:1347] Generated new cluster ID: a738ba3f15fa434495982601af43bfd6
I20260927 10:53:39.870005 20768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:39.877305 20768 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:39.878247 20768 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:39.884845 20768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae: Generated new TSK 0
I20260927 10:53:39.885705 20768 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.1.keytab.
Entry for principal oryx/127.14.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.1.keytab.
I20260927 10:53:39.918711 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:40605
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.1.keytab
--principal=oryx/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:46363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:40.029736 20775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:40.030035 20775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:40.030112 20775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:40.034158 20775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:40.034258 20775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:40.034368 20775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:53:40.039566 20775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:46363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.1.keytab
--principal=oryx/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:40605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:40.040644 20775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:40.041668 20775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:40.051312 20780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:40.051323 20783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:40.051409 20781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:40.051461 20775 server_base.cc:1034] running on GCE node
I20260927 10:53:40.052060 20775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:40.052702 20775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:40.053853 20775 hybrid_clock.cc:648] HybridClock initialized: now 1790506420053836 us; error 31 us; skew 500 ppm
Sep 27 10:53:40 dist-test-slave-tz51 krb5kdc[20668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506420, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:40.056593 20775 init.cc:377] Logged in from keytab as oryx/127.14.68.1@KRBTEST.COM (short username oryx)
I20260927 10:53:40.057498 20775 webserver.cc:460] Webserver started at http://127.14.68.1:44933/ using document root <none> and password file <none>
I20260927 10:53:40.058054 20775 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:40.058144 20775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:40.058379 20775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:40.060557 20775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "8f96cff0ec274e9eac28349a5a876141"
format_stamp: "Formatted at 2026-09-27 10:53:40 on dist-test-slave-tz51"
server_key: "4746ab0954b2ae9d576690a134cfb83c"
server_key_iv: "4e0b2302919ffc25e86d4ed69c5d395c"
server_key_version: "encryptionkey@0"
I20260927 10:53:40.061098 20775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "8f96cff0ec274e9eac28349a5a876141"
format_stamp: "Formatted at 2026-09-27 10:53:40 on dist-test-slave-tz51"
server_key: "4746ab0954b2ae9d576690a134cfb83c"
server_key_iv: "4e0b2302919ffc25e86d4ed69c5d395c"
server_key_version: "encryptionkey@0"
I20260927 10:53:40.064749 20775 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:53:40.067227 20790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:40.068554 20775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:53:40.068696 20775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "8f96cff0ec274e9eac28349a5a876141"
format_stamp: "Formatted at 2026-09-27 10:53:40 on dist-test-slave-tz51"
server_key: "4746ab0954b2ae9d576690a134cfb83c"
server_key_iv: "4e0b2302919ffc25e86d4ed69c5d395c"
server_key_version: "encryptionkey@0"
I20260927 10:53:40.068800 20775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:40.083877 20775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:40.086992 20775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:40.087193 20775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:40.087841 20775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:40.088660 20775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:40.088704 20775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:40.088770 20775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:40.088801 20775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:40.097635 20775 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:36925
I20260927 10:53:40.097651 20903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:36925 every 8 connection(s)
I20260927 10:53:40.098639 20775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:53:40.105608 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 20775
I20260927 10:53:40.105718 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:53:40.106046 14608 external_mini_cluster.cc:1428] Setting key 6d6c81237e9884b77d4cba8b1ee59216
Sep 27 10:53:40 dist-test-slave-tz51 krb5kdc[20668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506420, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM
I20260927 10:53:40.114046 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:40.100516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:35705 (local address 127.14.68.62:40605)
0927 10:53:40.100654 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:40.100657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:40.101532 (+ 875us) server_negotiation.cc:407] Connection header received
0927 10:53:40.102941 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:40.102944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:40.103004 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:40.103115 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:40.104549 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.105125 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.106491 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.106818 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.106940 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:40.110034 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:40.110037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:40.110039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:40.110064 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:40.111756 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:40.112407 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:40.112409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:40.112411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:40.112488 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:40.112911 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:40.112913 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:40.112914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:40.113084 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:40.113182 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:40.113820 (+ 638us) server_negotiation.cc:299] Negotiation successful
0927 10:53:40.113922 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:53:40.115535 20906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:40.100854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.1:35705)
0927 10:53:40.101358 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:40.101404 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:40.102722 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:40.103297 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:40.103304 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:40.103768 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:40.104321 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:40.104336 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.105262 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.105266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:40.106361 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:40.106372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.106977 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.106981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:40.107207 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:40.107911 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:40.107928 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:40.109829 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:40.111879 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:40.111885 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:40.111906 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:40.112304 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:40.112584 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:40.112589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:40.112593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:40.112794 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:40.113194 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:40.113202 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:40.113546 (+ 344us) client_negotiation.cc:769] Sending connection context
0927 10:53:40.113778 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 10:53:40.114211 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for oryx/127.14.68.2@KRBTEST.COM; defaulting to no policy
I20260927 10:53:40.116920 20904 heartbeater.cc:347] Connected to a master server at 127.14.68.62:40605
I20260927 10:53:40.117210 20904 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:40.117731 20904 heartbeater.cc:511] Master 127.14.68.62:40605 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.68.2@KRBTEST.COM" created.
I20260927 10:53:40.119463 20713 ts_manager.cc:194] Registered new tserver with Master: 8f96cff0ec274e9eac28349a5a876141 (127.14.68.1:36925)
I20260927 10:53:40.120910 20713 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:35705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.2.keytab.
Entry for principal oryx/127.14.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.2.keytab.
I20260927 10:53:40.135838 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:40605
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.2.keytab
--principal=oryx/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:46363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:40.245370 20909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:40.245700 20909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:40.245779 20909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:40.250198 20909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:40.250290 20909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:40.250379 20909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:53:40.255919 20909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:46363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.2.keytab
--principal=oryx/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:40605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:40.256994 20909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:40.258054 20909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:40.267987 20917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:40.268000 20914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:40.268005 20915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:40.268265 20909 server_base.cc:1034] running on GCE node
I20260927 10:53:40.268726 20909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:40.269339 20909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:40.270499 20909 hybrid_clock.cc:648] HybridClock initialized: now 1790506420270466 us; error 51 us; skew 500 ppm
Sep 27 10:53:40 dist-test-slave-tz51 krb5kdc[20668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506420, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:40.273339 20909 init.cc:377] Logged in from keytab as oryx/127.14.68.2@KRBTEST.COM (short username oryx)
I20260927 10:53:40.274230 20909 webserver.cc:460] Webserver started at http://127.14.68.2:36703/ using document root <none> and password file <none>
I20260927 10:53:40.274789 20909 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:40.274859 20909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:40.275089 20909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:40.277166 20909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "0a59c8b46a2b48c185bfc0cd886caa51"
format_stamp: "Formatted at 2026-09-27 10:53:40 on dist-test-slave-tz51"
server_key: "891aca288ee44679ad4b33c608f56e2f"
server_key_iv: "72417871db0e865fd568f108b56e9c11"
server_key_version: "encryptionkey@0"
I20260927 10:53:40.277720 20909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "0a59c8b46a2b48c185bfc0cd886caa51"
format_stamp: "Formatted at 2026-09-27 10:53:40 on dist-test-slave-tz51"
server_key: "891aca288ee44679ad4b33c608f56e2f"
server_key_iv: "72417871db0e865fd568f108b56e9c11"
server_key_version: "encryptionkey@0"
I20260927 10:53:40.281435 20909 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:53:40.283948 20924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:40.285233 20909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:40.285395 20909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "0a59c8b46a2b48c185bfc0cd886caa51"
format_stamp: "Formatted at 2026-09-27 10:53:40 on dist-test-slave-tz51"
server_key: "891aca288ee44679ad4b33c608f56e2f"
server_key_iv: "72417871db0e865fd568f108b56e9c11"
server_key_version: "encryptionkey@0"
I20260927 10:53:40.285513 20909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:40.296916 20909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:40.300321 20909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:40.300516 20909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:40.301085 20909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:40.301940 20909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:40.302008 20909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:40.302078 20909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:40.302114 20909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:40.311179 20909 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:43831
I20260927 10:53:40.311233 21037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:43831 every 8 connection(s)
I20260927 10:53:40.312137 20909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
Sep 27 10:53:40 dist-test-slave-tz51 krb5kdc[20668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506420, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM
I20260927 10:53:40.322124 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 20909
I20260927 10:53:40.322244 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:53:40.322546 14608 external_mini_cluster.cc:1428] Setting key a330e002a4ce6c53876119ec22df4405
I20260927 10:53:40.325841 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:40.313965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:39821 (local address 127.14.68.62:40605)
0927 10:53:40.314147 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:40.314151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:40.314921 (+ 770us) server_negotiation.cc:407] Connection header received
0927 10:53:40.316225 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:40.316228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:40.316288 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:40.316385 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:40.317546 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.318089 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.319235 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.319464 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.319578 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:40.321996 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:40.321999 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:40.322001 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:40.322024 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:40.323554 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:40.324230 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:40.324233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:40.324234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:40.324310 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:40.324742 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:40.324744 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:40.324745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:40.324916 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:40.324989 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:40.325630 (+ 641us) server_negotiation.cc:299] Negotiation successful
0927 10:53:40.325730 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:53:40.326654 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:40.314300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.2:39821)
0927 10:53:40.314769 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:40.314810 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:40.316017 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:40.316552 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:40.316559 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:40.316936 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:40.317404 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:40.317417 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.318221 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.318229 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:40.319109 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:40.319119 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.319617 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.319621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:40.319839 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:40.320371 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:40.320386 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:40.321806 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:40.323678 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:40.323684 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:40.323720 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:40.324052 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:40.324433 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:40.324438 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:40.324443 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:40.324636 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:40.325026 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:40.325035 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:40.325340 (+ 305us) client_negotiation.cc:769] Sending connection context
0927 10:53:40.325537 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 10:53:40.325859 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":110,"threads_started":1}
I20260927 10:53:40.327762 21038 heartbeater.cc:347] Connected to a master server at 127.14.68.62:40605
I20260927 10:53:40.327988 21038 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:40.328528 21038 heartbeater.cc:511] Master 127.14.68.62:40605 requested a full tablet report, sending...
I20260927 10:53:40.329699 20713 ts_manager.cc:194] Registered new tserver with Master: 0a59c8b46a2b48c185bfc0cd886caa51 (127.14.68.2:43831)
I20260927 10:53:40.330267 20713 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:39821
WARNING: no policy specified for oryx/127.14.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.3.keytab.
Entry for principal oryx/127.14.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.3.keytab.
I20260927 10:53:40.350802 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:40605
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.3.keytab
--principal=oryx/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:46363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:40.460649 21043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:40.460951 21043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:40.461024 21043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:40.465054 21043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:40.465133 21043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:40.465224 21043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:53:40.470498 21043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:46363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.3.keytab
--principal=oryx/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:40605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:40.471549 21043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:40.472625 21043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:40.482131 21048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:40.482173 21049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:40.482126 21051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:40.482512 21043 server_base.cc:1034] running on GCE node
I20260927 10:53:40.482847 21043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:40.483474 21043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:40.484632 21043 hybrid_clock.cc:648] HybridClock initialized: now 1790506420484600 us; error 41 us; skew 500 ppm
Sep 27 10:53:40 dist-test-slave-tz51 krb5kdc[20668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506420, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:40.487306 21043 init.cc:377] Logged in from keytab as oryx/127.14.68.3@KRBTEST.COM (short username oryx)
I20260927 10:53:40.488204 21043 webserver.cc:460] Webserver started at http://127.14.68.3:38207/ using document root <none> and password file <none>
I20260927 10:53:40.488741 21043 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:40.488827 21043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:40.489046 21043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:40.491058 21043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "198bfa5be48e491eb86d59eab104046e"
format_stamp: "Formatted at 2026-09-27 10:53:40 on dist-test-slave-tz51"
server_key: "14bfdda7da8603d8d54a759740786c47"
server_key_iv: "fcf81306d899c9967ba9cea69a53c7c6"
server_key_version: "encryptionkey@0"
I20260927 10:53:40.491629 21043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "198bfa5be48e491eb86d59eab104046e"
format_stamp: "Formatted at 2026-09-27 10:53:40 on dist-test-slave-tz51"
server_key: "14bfdda7da8603d8d54a759740786c47"
server_key_iv: "fcf81306d899c9967ba9cea69a53c7c6"
server_key_version: "encryptionkey@0"
I20260927 10:53:40.495167 21043 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:53:40.497685 21058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:40.498886 21043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:40.499034 21043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "198bfa5be48e491eb86d59eab104046e"
format_stamp: "Formatted at 2026-09-27 10:53:40 on dist-test-slave-tz51"
server_key: "14bfdda7da8603d8d54a759740786c47"
server_key_iv: "fcf81306d899c9967ba9cea69a53c7c6"
server_key_version: "encryptionkey@0"
I20260927 10:53:40.499168 21043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:40.511029 21043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:40.513564 21043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:40.513700 21043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:40.514178 21043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:40.514849 21043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:40.514892 21043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:40.514927 21043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:40.514947 21043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:40.523377 21043 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:36575
I20260927 10:53:40.523401 21171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:36575 every 8 connection(s)
I20260927 10:53:40.524381 21043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:53:40.526991 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 21043
I20260927 10:53:40.527127 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:53:40.527458 14608 external_mini_cluster.cc:1428] Setting key 3e95f78df0ac29f2ff605fbd6a52466d
Sep 27 10:53:40 dist-test-slave-tz51 krb5kdc[20668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506420, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM
I20260927 10:53:40.539870 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:40.527886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:44765 (local address 127.14.68.62:40605)
0927 10:53:40.528050 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:40.528142 (+ 92us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:40.528613 (+ 471us) server_negotiation.cc:407] Connection header received
0927 10:53:40.530325 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:40.530328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:40.530378 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:40.530448 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:40.531665 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.532266 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.533474 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.533687 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.533771 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:40.536523 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:40.536526 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:40.536528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:40.536554 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:40.538024 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:40.538488 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:40.538491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:40.538492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:40.538563 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:40.538868 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:40.538870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:40.538871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:40.539040 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:40.539113 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:40.539570 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 10:53:40.539674 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 10:53:40.541075 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:40.527412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.3:44765)
0927 10:53:40.528412 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:40.528470 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:40.530115 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:40.530594 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:40.530601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:40.530999 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:40.531520 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:40.531535 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.532392 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.532395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:40.533357 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:40.533367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.533979 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.533982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:40.534171 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:40.534760 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:40.534776 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:40.536350 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:40.538129 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:40.538134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:40.538146 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:40.538401 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:40.538640 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:40.538642 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:40.538643 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:40.538789 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:40.539124 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:40.539129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:40.539367 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 10:53:40.539525 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 10:53:40.539882 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":179,"threads_started":1}
I20260927 10:53:40.542191 21172 heartbeater.cc:347] Connected to a master server at 127.14.68.62:40605
I20260927 10:53:40.542418 21172 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:40.542889 21172 heartbeater.cc:511] Master 127.14.68.62:40605 requested a full tablet report, sending...
I20260927 10:53:40.543901 20713 ts_manager.cc:194] Registered new tserver with Master: 198bfa5be48e491eb86d59eab104046e (127.14.68.3:36575)
I20260927 10:53:40.544438 20713 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:44765
I20260927 10:53:40.553503 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.68.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.68.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.68.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.61.keytab.
Entry for principal oryx/127.14.68.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.61.keytab.
I20260927 10:53:40.580052 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.61:33935
--webserver_interface=127.14.68.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.61.keytab
--principal=oryx/127.14.68.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:46363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:40605,127.14.68.61:33935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:40.690178 21177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:40.690469 21177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:40.690565 21177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:40.694581 21177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:40.694667 21177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:40.694706 21177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:40.694763 21177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:40.694808 21177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:40.700815 21177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:46363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:40605,127.14.68.61:33935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.61.keytab
--principal=oryx/127.14.68.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.61:33935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.68.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
W20260927 10:53:40.701076 21177 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.14.68.62:40605,127.14.68.61:33935'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 10:53:40.701972 21177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:40.703063 21177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:53:40.711658 21177 server_base.cc:1034] running on GCE node
W20260927 10:53:40.711688 21183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:40.711876 21185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:40.711673 21182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:40.712558 21177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:40.713603 21177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:40.714819 21177 hybrid_clock.cc:648] HybridClock initialized: now 1790506420714810 us; error 33 us; skew 500 ppm
Sep 27 10:53:40 dist-test-slave-tz51 krb5kdc[20668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506420, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:40.717489 21177 init.cc:377] Logged in from keytab as oryx/127.14.68.61@KRBTEST.COM (short username oryx)
I20260927 10:53:40.718327 21177 webserver.cc:460] Webserver started at http://127.14.68.61:45549/ using document root <none> and password file <none>
I20260927 10:53:40.718881 21177 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:40.718971 21177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:40.719200 21177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:40.721251 21177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/data/instance:
uuid: "cf60ff31248349aea2c39fa73946403f"
format_stamp: "Formatted at 2026-09-27 10:53:40 on dist-test-slave-tz51"
server_key: "b3872e6ec372a79f32a84047fbf5f33b"
server_key_iv: "0d58f1fb52469111f25a9e035a48aff7"
server_key_version: "encryptionkey@0"
I20260927 10:53:40.721810 21177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/wal/instance:
uuid: "cf60ff31248349aea2c39fa73946403f"
format_stamp: "Formatted at 2026-09-27 10:53:40 on dist-test-slave-tz51"
server_key: "b3872e6ec372a79f32a84047fbf5f33b"
server_key_iv: "0d58f1fb52469111f25a9e035a48aff7"
server_key_version: "encryptionkey@0"
I20260927 10:53:40.725502 21177 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 10:53:40.727871 21192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:40.729081 21177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:53:40.729221 21177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/wal
uuid: "cf60ff31248349aea2c39fa73946403f"
format_stamp: "Formatted at 2026-09-27 10:53:40 on dist-test-slave-tz51"
server_key: "b3872e6ec372a79f32a84047fbf5f33b"
server_key_iv: "0d58f1fb52469111f25a9e035a48aff7"
server_key_version: "encryptionkey@0"
I20260927 10:53:40.729333 21177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:40.749022 21177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:40.752031 21177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:40.752246 21177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:40.759514 21177 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.61:33935
I20260927 10:53:40.759878 21244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.61:33935 every 8 connection(s)
I20260927 10:53:40.760667 21177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/data/info.pb
I20260927 10:53:40.763394 21245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:40.766403 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 21177
I20260927 10:53:40.767499 21245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:53:40 dist-test-slave-tz51 krb5kdc[20668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506420, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.61@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM
Sep 27 10:53:40 dist-test-slave-tz51 krb5kdc[20668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.68.61@KRBTEST.COM
I20260927 10:53:40.780683 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:40.769383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40760 (local address 127.14.68.62:40605)
0927 10:53:40.769515 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:40.769519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:40.769824 (+ 305us) server_negotiation.cc:407] Connection header received
0927 10:53:40.770045 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:40.770048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:40.770100 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:40.770193 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:40.771077 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.771933 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.773412 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.773744 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.773842 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:40.777024 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:40.777028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:40.777030 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:40.777055 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:40.778656 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:40.779139 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:40.779143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:40.779145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:40.779227 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:40.779553 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:40.779557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:40.779559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:40.779876 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:40.779969 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:40.780390 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 10:53:40.780510 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:53:40.781371 21248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:40.767920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39686 (local address 127.14.68.61:33935)
0927 10:53:40.768359 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:40.768375 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:40.768410 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:53:40.769110 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:40.769133 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:40.769542 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:40.769858 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:40.770725 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.772100 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.773244 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.773730 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.773889 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:40.777014 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:40.777019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:40.777028 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:40.777048 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:40.778619 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:40.779050 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:40.779061 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:40.779062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:40.779160 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:40.779435 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:40.779438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:40.779440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:40.779844 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:40.780054 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:40.780256 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 10:53:40.780653 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":298,"spinlock_wait_cycles":2432,"thread_start_us":143,"threads_started":1}
I20260927 10:53:40.781371 21249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:40.769309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.0.0.1:40760)
0927 10:53:40.769684 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:40.769719 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:40.769921 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:40.770276 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:40.770283 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:40.770735 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:40.770928 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:40.770947 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.772063 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.772065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:40.773297 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:40.773314 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.773891 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.773895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:40.774005 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:40.774603 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:40.774626 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:40.776870 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:40.778787 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:40.778802 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:40.778811 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:40.779035 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:40.779341 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:40.779344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:40.779346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:40.779458 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:40.780019 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:40.780026 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:40.780171 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 10:53:40.780334 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 10:53:40.780579 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":31,"thread_start_us":123,"threads_started":1}
I20260927 10:53:40.782676 21245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:53:40 dist-test-slave-tz51 krb5kdc[20668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506420, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.61@KRBTEST.COM for oryx/127.14.68.61@KRBTEST.COM
I20260927 10:53:40.790079 21249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:40.783159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:33935 (local address 127.0.0.1:39692)
0927 10:53:40.783256 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:40.783269 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:40.783339 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:40.783787 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:40.783789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:40.783912 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:40.783992 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:40.783996 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.784655 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.784657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:40.785152 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:40.785159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.785543 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.785546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:40.785613 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:40.786051 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:40.786071 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:40.787235 (+ 1164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:40.788820 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:40.788822 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:40.788823 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:40.789042 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:40.789307 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:40.789309 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:40.789311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:40.789391 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:40.789695 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:40.789697 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:40.789768 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 10:53:40.789848 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 10:53:40.789949 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 10:53:40.790257 21248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:40.783299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39692 (local address 127.14.68.61:33935)
0927 10:53:40.783580 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:40.783583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:40.783594 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:53:40.783619 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:40.783621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:40.783652 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:40.783772 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:40.784082 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.784567 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.785243 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.785458 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.785529 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:40.787353 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:40.787356 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:40.787358 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:40.787377 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:40.788732 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:40.789130 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:40.789133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:40.789134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:40.789205 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:40.789471 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:40.789472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:40.789473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:40.789614 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:40.789683 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:40.789878 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 10:53:40.789999 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":228}
I20260927 10:53:40.793275 21245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f: Bootstrap starting.
I20260927 10:53:40.795475 21245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:40.796255 21245 log.cc:824] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:40.798032 21245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f: No bootstrap required, opened a new log
I20260927 10:53:40.801095 21245 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } }
I20260927 10:53:40.801321 21245 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:40.801409 21245 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf60ff31248349aea2c39fa73946403f, State: Initialized, Role: FOLLOWER
I20260927 10:53:40.801839 21245 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } }
I20260927 10:53:40.802335 21251 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } } }
I20260927 10:53:40.802440 21251 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:53:40.803027 21245 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [sys.catalog]: configured and running, proceeding with master startup.
W20260927 10:53:40.807482 21262 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:53:40.807570 21262 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:53:40.809947 21177 master_runner.cc:428] Detected that this master cf60ff31248349aea2c39fa73946403f is joining an existing cluster
I20260927 10:53:40.810007 21177 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.68.61:33935
I20260927 10:53:40.822526 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:40.812137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.0.0.1:40764)
0927 10:53:40.813150 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:40.813167 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:40.813278 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:40.813602 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:40.813608 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:40.813906 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:40.814043 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:40.814050 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.815140 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.815143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:40.816050 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:40.816058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.816737 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.816740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:40.816837 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:40.817389 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:40.817402 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:40.818157 (+ 755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:40.820555 (+ 2398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:40.820558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:40.820568 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:40.820982 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:40.821297 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:40.821300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:40.821302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:40.821400 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:40.821899 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:40.821902 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:40.822056 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 10:53:40.822197 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 10:53:40.822331 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":915,"mutex_wait_us":28,"thread_start_us":102,"threads_started":1}
I20260927 10:53:40.822566 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:40.812272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40764 (local address 127.14.68.62:40605)
0927 10:53:40.812466 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:40.812473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:40.813272 (+ 799us) server_negotiation.cc:407] Connection header received
0927 10:53:40.813385 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:40.813389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:40.813460 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:40.813547 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:40.814183 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.815017 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.816173 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.816503 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.816573 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:40.818332 (+ 1759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:40.818337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:40.818342 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:40.818375 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:40.820435 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:40.821095 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:40.821098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:40.821100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:40.821197 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:40.821518 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:40.821522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:40.821524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:40.821754 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:40.821826 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:40.822266 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 10:53:40.822404 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 10:53:40.822660 21248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:40.812812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39694 (local address 127.14.68.61:33935)
0927 10:53:40.812960 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:40.812964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:40.814119 (+ 1155us) server_negotiation.cc:407] Connection header received
0927 10:53:40.814254 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:40.814258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:40.814290 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:40.814346 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:40.814889 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.815608 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.816722 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.817044 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.817119 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:40.819260 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:40.819264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:40.819268 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:40.819305 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:40.821002 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:40.821475 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:40.821480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:40.821482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:40.821557 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:40.821816 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:40.821821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:40.821823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:40.822028 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:40.822129 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:40.822381 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 10:53:40.822475 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":57}
I20260927 10:53:40.822922 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:40.812741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:33935 (local address 127.0.0.1:39694)
0927 10:53:40.814048 (+ 1307us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:40.814062 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:40.814173 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:40.814449 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:40.814452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:40.814663 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:40.814781 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:40.814787 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.815765 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.815767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:40.816613 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:40.816622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.817261 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.817265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:40.817386 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:40.818673 (+ 1287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:40.818688 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:40.819113 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:40.821131 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:40.821136 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:40.821141 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:40.821380 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:40.821643 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:40.821647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:40.821649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:40.821729 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:40.822119 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:40.822122 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:40.822200 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 10:53:40.822550 (+ 350us) client_negotiation.cc:240] Negotiation successful
0927 10:53:40.822798 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1242,"thread_start_us":115,"threads_started":1}
Sep 27 10:53:40 dist-test-slave-tz51 krb5kdc[20668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506419, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.62@KRBTEST.COM for oryx/127.14.68.61@KRBTEST.COM
I20260927 10:53:40.838809 21248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:40.827436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39700 (local address 127.14.68.61:33935)
0927 10:53:40.827556 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:40.827566 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:40.827889 (+ 323us) server_negotiation.cc:407] Connection header received
0927 10:53:40.828070 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:40.828073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:40.828123 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:40.828210 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:40.828961 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.829856 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.831115 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.831344 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.831452 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:40.833992 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:40.833995 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:40.833998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:40.834028 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:40.836952 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:40.837474 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:40.837477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:40.837479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:40.837553 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:40.837830 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:40.837833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:40.837835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:40.838029 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:40.838096 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:40.838528 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 10:53:40.838637 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:53:40.839293 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:40.827435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:33935 (local address 127.0.0.1:39700)
0927 10:53:40.827761 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:40.827812 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:40.827941 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:40.828247 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:40.828252 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:40.828637 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:40.828823 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:40.828830 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.829980 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.829985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:40.830992 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:40.831002 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.831455 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.831459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:40.831575 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:40.832145 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:40.832158 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:40.833876 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:40.837071 (+ 3195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:40.837080 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:40.837086 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:40.837367 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:40.837650 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:40.837652 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:40.837654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:40.837728 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:40.838191 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:40.838196 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:40.838316 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 10:53:40.838938 (+ 622us) client_negotiation.cc:240] Negotiation successful
0927 10:53:40.839139 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":71,"threads_started":1}
I20260927 10:53:40.840749 20713 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.68.61:33935: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cf60ff31248349aea2c39fa73946403f"
member_type: NON_VOTER
last_known_addr {
host: "127.14.68.61"
port: 33935
}
attrs {
promote: true
}
}
dest_uuid: "e37e387fd038459890cc0553983720ae"
cas_config_opid_index: -1
I20260927 10:53:40.841424 20713 consensus_queue.cc:237] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: NON_VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: true } }
I20260927 10:53:40.844336 21273 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER cf60ff31248349aea2c39fa73946403f (127.14.68.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: NON_VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: true } } }
I20260927 10:53:40.844756 21219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:40.846112 21273 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.68.61:33935
I20260927 10:53:40.846205 21219 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 1 FOLLOWER]: Refusing update from remote peer e37e387fd038459890cc0553983720ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 10:53:40.846383 21273 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e37e387fd038459890cc0553983720ae" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: NON_VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: true } } }
I20260927 10:53:40.846468 21273 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:40.846715 21177 master.cc:537] Master@127.14.68.61:33935 shutting down...
I20260927 10:53:40.847440 21273 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: NON_VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 10:53:40.849196 20697 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 10:53:40.851241 21177 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:40.851594 21177 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:40.851787 21177 tablet_replica.cc:330] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f: stopping tablet replica
WARNING: no policy specified for oryx/127.14.68.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.68.60@KRBTEST.COM" created.
I20260927 10:53:40.858047 21177 master.cc:552] Master@127.14.68.61:33935 shutdown complete.
I20260927 10:53:40.858104 21177 master_runner.cc:305] Clearing existing system tablet
I20260927 10:53:40.858376 21177 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:40.858436 21177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:40.859546 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:40.859223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:33935 (local address 127.0.0.1:39710)
0927 10:53:40.859399 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:40.859464 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.61:33935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260927 10:53:40.859952 20697 consensus_peers.cc:487] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae -> Peer cf60ff31248349aea2c39fa73946403f (127.14.68.61:33935): Couldn't send request to peer cf60ff31248349aea2c39fa73946403f. Status: Network error: Client connection negotiation failed: client connection to 127.14.68.61:33935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 10:53:40.863302 21177 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260927 10:53:40.864293 21278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:40.864640 21177 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:40.865106 21177 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:53:40.865178 21177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/wal
uuid: "cf60ff31248349aea2c39fa73946403f"
format_stamp: "Formatted at 2026-09-27 10:53:40 on dist-test-slave-tz51"
server_key: "b3872e6ec372a79f32a84047fbf5f33b"
server_key_iv: "0d58f1fb52469111f25a9e035a48aff7"
server_key_version: "encryptionkey@0"
I20260927 10:53:40.867115 21177 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:40.869537 21177 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 10:53:40.869685 21177 log.cc:1196] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 10:53:40.870211 21177 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f: Deleting consensus metadata
I20260927 10:53:40.870401 21177 master_runner.cc:315] Copying system tablet from 127.14.68.62:40605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.68.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.60.keytab.
Entry for principal oryx/127.14.68.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.60.keytab.
I20260927 10:53:40.871542 21177 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.68.62:40605
I20260927 10:53:40.871569 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.60:35011
--webserver_interface=127.14.68.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.60.keytab
--principal=oryx/127.14.68.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:46363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:40605,127.14.68.61:33935,127.14.68.60:35011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:53:40.880888 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:40.872320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40772 (local address 127.14.68.62:40605)
0927 10:53:40.872452 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:40.872459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:40.872694 (+ 235us) server_negotiation.cc:407] Connection header received
0927 10:53:40.872854 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:40.872857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:40.872929 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:40.873005 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:40.874013 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.874852 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.875746 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.875985 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.876101 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:40.877262 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:40.877265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:40.877268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:40.877304 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:40.878983 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:40.879575 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:40.879581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:40.879582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:40.879678 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:40.880017 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:40.880021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:40.880022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:40.880220 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:40.880329 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:40.880600 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 10:53:40.880712 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:53:40.880903 21284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:40.872260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.0.0.1:40772)
0927 10:53:40.872593 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:40.872605 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:40.872745 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:40.873021 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:40.873024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:40.873321 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:40.873839 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:40.873853 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.875028 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.875031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:40.875605 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:40.875612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:40.876104 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:40.876108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:40.876200 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:40.876772 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:40.876784 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:40.877108 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:40.879098 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:40.879102 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:40.879105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:40.879337 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:40.879810 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:40.879814 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:40.879816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:40.879914 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:40.880317 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:40.880320 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:40.880431 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 10:53:40.880537 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 10:53:40.880712 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":91,"threads_started":1}
I20260927 10:53:40.881615 20733 tablet_copy_service.cc:140] P e37e387fd038459890cc0553983720ae: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cf60ff31248349aea2c39fa73946403f ({username='oryx', principal='oryx/127.14.68.61@KRBTEST.COM'} at 127.0.0.1:40772)
I20260927 10:53:40.881804 20733 tablet_copy_service.cc:161] P e37e387fd038459890cc0553983720ae: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cf60ff31248349aea2c39fa73946403f at {username='oryx', principal='oryx/127.14.68.61@KRBTEST.COM'} at 127.0.0.1:40772: session id = cf60ff31248349aea2c39fa73946403f-00000000000000000000000000000000
I20260927 10:53:40.883220 20733 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 10:53:40.884738 21177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 10:53:40.886613 21177 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f: tablet copy: Starting download of 0 data blocks...
I20260927 10:53:40.886824 21177 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f: tablet copy: Starting download of 1 WAL segments...
I20260927 10:53:40.888386 21177 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 10:53:40.889977 20733 tablet_copy_service.cc:342] P e37e387fd038459890cc0553983720ae: Request end of tablet copy session cf60ff31248349aea2c39fa73946403f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.68.61@KRBTEST.COM'} at 127.0.0.1:40772
I20260927 10:53:40.890101 20733 tablet_copy_service.cc:433] P e37e387fd038459890cc0553983720ae: ending tablet copy session cf60ff31248349aea2c39fa73946403f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cf60ff31248349aea2c39fa73946403f
I20260927 10:53:40.896767 21177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:40.898622 21286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:40.898661 21289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:40.898900 21287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:40.898976 21177 server_base.cc:1034] running on GCE node
I20260927 10:53:40.899224 21177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:40.899582 21177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:40.900720 21177 hybrid_clock.cc:648] HybridClock initialized: now 1790506420900706 us; error 32 us; skew 500 ppm
Sep 27 10:53:40 dist-test-slave-tz51 krb5kdc[20668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506420, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:40.902258 21177 init.cc:377] Logged in from keytab as oryx/127.14.68.61@KRBTEST.COM (short username oryx)
I20260927 10:53:40.902735 21177 webserver.cc:460] Webserver started at http://127.14.68.61:34391/ using document root <none> and password file <none>
I20260927 10:53:40.902931 21177 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:40.902987 21177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:40.904323 21177 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 10:53:40.905221 21296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:40.905452 21177 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 10:53:40.905543 21177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/wal
uuid: "cf60ff31248349aea2c39fa73946403f"
format_stamp: "Formatted at 2026-09-27 10:53:40 on dist-test-slave-tz51"
server_key: "b3872e6ec372a79f32a84047fbf5f33b"
server_key_iv: "0d58f1fb52469111f25a9e035a48aff7"
server_key_version: "encryptionkey@0"
I20260927 10:53:40.905642 21177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:40.914090 21177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:40.916193 21177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:40.921046 21177 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.61:33935
I20260927 10:53:40.921617 21177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-1/data/info.pb
I20260927 10:53:40.922973 21349 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:53:40.923846 21349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f: Bootstrap starting.
I20260927 10:53:40.927842 21348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.61:33935 every 8 connection(s)
I20260927 10:53:40.931770 21349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:53:40.932044 21349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f: Bootstrap complete.
I20260927 10:53:40.932502 21349 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: NON_VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: true } }
I20260927 10:53:40.932641 21349 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf60ff31248349aea2c39fa73946403f, State: Initialized, Role: LEARNER
I20260927 10:53:40.932776 21349 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: NON_VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: true } }
I20260927 10:53:40.933079 21352 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: NON_VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: true } } }
I20260927 10:53:40.933239 21349 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:40.933239 21352 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [sys.catalog]: This master's current role is: LEARNER
I20260927 10:53:40.936352 21363 catalog_manager.cc:1259] Loaded cluster ID: a738ba3f15fa434495982601af43bfd6
I20260927 10:53:40.936452 21363 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f: loading cluster ID for follower catalog manager: success
I20260927 10:53:40.937919 21363 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f: acquiring CA information for follower catalog manager: success
I20260927 10:53:40.938977 21363 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 10:53:41.030076 21285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:41.030388 21285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:41.030429 21285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:41.034523 21285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:41.034587 21285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:41.034606 21285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:41.034669 21285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:41.034700 21285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:41.040037 21285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:46363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:40605,127.14.68.61:33935,127.14.68.60:35011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.60.keytab
--principal=oryx/127.14.68.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.60:35011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.68.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:41.041147 21285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:41.042325 21285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:41.051308 21368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:41.051370 21285 server_base.cc:1034] running on GCE node
W20260927 10:53:41.051318 21371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:41.051383 21369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:41.052183 21285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:41.053264 21285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:41.054425 21285 hybrid_clock.cc:648] HybridClock initialized: now 1790506421054392 us; error 47 us; skew 500 ppm
Sep 27 10:53:41 dist-test-slave-tz51 krb5kdc[20668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506421, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:41.057128 21285 init.cc:377] Logged in from keytab as oryx/127.14.68.60@KRBTEST.COM (short username oryx)
I20260927 10:53:41.057978 21285 webserver.cc:460] Webserver started at http://127.14.68.60:35685/ using document root <none> and password file <none>
I20260927 10:53:41.058585 21285 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:41.058700 21285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:41.058938 21285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:41.061124 21285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/data/instance:
uuid: "fb6e814658944ac489f240027c3e09ba"
format_stamp: "Formatted at 2026-09-27 10:53:41 on dist-test-slave-tz51"
server_key: "cdc4c326da93fbdf3a372368155aabc4"
server_key_iv: "17926e42762bf471ab5a9b220c8d1970"
server_key_version: "encryptionkey@0"
I20260927 10:53:41.061681 21285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/wal/instance:
uuid: "fb6e814658944ac489f240027c3e09ba"
format_stamp: "Formatted at 2026-09-27 10:53:41 on dist-test-slave-tz51"
server_key: "cdc4c326da93fbdf3a372368155aabc4"
server_key_iv: "17926e42762bf471ab5a9b220c8d1970"
server_key_version: "encryptionkey@0"
I20260927 10:53:41.065313 21285 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:53:41.067647 21378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:41.068941 21285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:41.069083 21285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/wal
uuid: "fb6e814658944ac489f240027c3e09ba"
format_stamp: "Formatted at 2026-09-27 10:53:41 on dist-test-slave-tz51"
server_key: "cdc4c326da93fbdf3a372368155aabc4"
server_key_iv: "17926e42762bf471ab5a9b220c8d1970"
server_key_version: "encryptionkey@0"
I20260927 10:53:41.069202 21285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:41.084312 21285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:41.087119 21285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:41.087330 21285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:41.094584 21285 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.60:35011
I20260927 10:53:41.094887 21430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.60:35011 every 8 connection(s)
I20260927 10:53:41.095619 21285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/data/info.pb
I20260927 10:53:41.098661 21431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:41.101855 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 21285
I20260927 10:53:41.102907 21431 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:53:41 dist-test-slave-tz51 krb5kdc[20668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506421, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.60@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM
Sep 27 10:53:41 dist-test-slave-tz51 krb5kdc[20668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.68.60@KRBTEST.COM
I20260927 10:53:41.116629 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.105066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40782 (local address 127.14.68.62:40605)
0927 10:53:41.105306 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.105311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.105531 (+ 220us) server_negotiation.cc:407] Connection header received
0927 10:53:41.105716 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.105720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.105771 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.105848 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:41.107038 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.107861 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.109345 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.109704 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.109810 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.112615 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:41.112619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:41.112621 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:41.112650 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:41.114452 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.115032 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.115036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.115037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.115131 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.115478 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.115483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.115485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.115685 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:41.115821 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.116336 (+ 515us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.116462 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260927 10:53:41.117501 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.104880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.0.0.1:40782)
0927 10:53:41.105304 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:41.105413 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:41.105593 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:41.105968 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:41.105977 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:41.106501 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:41.106834 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.106846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.107977 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.107980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.109199 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.109210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.109848 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.109852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.110035 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.110670 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:41.110693 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:41.112466 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:41.114565 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.114572 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.114581 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.114933 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.115227 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.115230 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.115232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.115379 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.115819 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:41.115826 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:41.116094 (+ 268us) client_negotiation.cc:769] Sending connection context
0927 10:53:41.116294 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 10:53:41.116626 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":151,"threads_started":1}
I20260927 10:53:41.117501 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.103616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48030 (local address 127.14.68.60:35011)
0927 10:53:41.104075 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.104093 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.104137 (+ 44us) server_negotiation.cc:407] Connection header received
0927 10:53:41.104960 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.104984 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.105413 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.105730 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:41.106682 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.108046 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.109225 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.110041 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.110125 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.113141 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:41.113148 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:41.113163 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:41.113191 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:41.115023 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.115463 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.115469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.115471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.115553 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.115898 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.115901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.115903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.116160 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:41.116372 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.116593 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.117007 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":144,"threads_started":1}
I20260927 10:53:41.118844 21431 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:53:41 dist-test-slave-tz51 krb5kdc[20668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506421, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.60@KRBTEST.COM for oryx/127.14.68.61@KRBTEST.COM
I20260927 10:53:41.124095 20904 heartbeater.cc:503] Master 127.14.68.62:40605 was elected leader, sending a full tablet report...
I20260927 10:53:41.126982 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.119352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:33935 (local address 127.0.0.1:39726)
0927 10:53:41.119500 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:41.119514 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:41.119579 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:41.120016 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:41.120019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:41.120181 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:41.120301 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.120307 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.121194 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.121197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.121719 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.121726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.122121 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.122123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.122196 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.122614 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:41.122625 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:41.123847 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:41.125765 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.125768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.125769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.125950 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.126222 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.126224 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.126225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.126298 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.126610 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:41.126612 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:41.126683 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 10:53:41.126774 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 10:53:41.126871 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 10:53:41.127043 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.119471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39726 (local address 127.14.68.61:33935)
0927 10:53:41.119769 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.119781 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.119795 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:53:41.119838 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.119841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.119899 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.119975 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:41.120411 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.121068 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.121815 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.122025 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.122100 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.123991 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:41.123994 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:41.123996 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:41.124017 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:41.125665 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.126033 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.126036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.126037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.126119 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.126374 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.126376 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.126377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.126533 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:41.126606 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.126808 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.126897 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":89,"threads_started":1}
I20260927 10:53:41.127673 21431 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:53:41 dist-test-slave-tz51 krb5kdc[20668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506421, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.60@KRBTEST.COM for oryx/127.14.68.60@KRBTEST.COM
I20260927 10:53:41.134960 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.128154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:35011 (local address 127.0.0.1:48036)
0927 10:53:41.128262 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:41.128276 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:41.128348 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:41.128563 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:41.128565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:41.128685 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:41.128764 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.128769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.129439 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.129441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.129918 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.129924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.130388 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.130391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.130444 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.130876 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:41.130887 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:41.132068 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:41.133558 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.133565 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.133567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.133751 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.134012 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.134015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.134017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.134092 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.134455 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:41.134459 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:41.134540 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 10:53:41.134657 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 10:53:41.134784 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 10:53:41.134960 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.128309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48036 (local address 127.14.68.60:35011)
0927 10:53:41.128406 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.128409 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.128419 (+ 10us) server_negotiation.cc:407] Connection header received
0927 10:53:41.128445 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.128447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.128477 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.128537 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:41.128855 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.129346 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.130014 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.130296 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.130375 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.132218 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:41.132221 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:41.132223 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:41.132241 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:41.133461 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.133842 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.133848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.133850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.133920 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.134180 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.134185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.134187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.134348 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:41.134452 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.134699 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.134835 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":42}
I20260927 10:53:41.138141 21431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba: Bootstrap starting.
I20260927 10:53:41.140434 21431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:41.141148 21431 log.cc:824] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:41.142944 21431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba: No bootstrap required, opened a new log
I20260927 10:53:41.145893 21431 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } }
I20260927 10:53:41.146098 21431 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:41.146170 21431 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb6e814658944ac489f240027c3e09ba, State: Initialized, Role: FOLLOWER
I20260927 10:53:41.146567 21431 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } }
I20260927 10:53:41.147020 21437 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } } }
I20260927 10:53:41.147162 21437 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:53:41.147779 21431 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [sys.catalog]: configured and running, proceeding with master startup.
W20260927 10:53:41.152370 21448 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:53:41.152451 21448 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:53:41.155488 21285 master_runner.cc:428] Detected that this master fb6e814658944ac489f240027c3e09ba is joining an existing cluster
I20260927 10:53:41.155553 21285 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.68.60:35011
I20260927 10:53:41.171072 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 20775
I20260927 10:53:41.177280 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:36925
--local_ip_for_outbound_sockets=127.14.68.1
--tserver_master_addrs=127.14.68.62:40605
--webserver_port=44933
--webserver_interface=127.14.68.1
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.1.keytab
--principal=oryx/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:46363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.68.62:40605,127.14.68.61:33935
--tserver_master_addrs=127.14.68.62:40605,127.14.68.61:33935,127.14.68.60:35011 with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:53:41.185714 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.157548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.0.0.1:40796)
0927 10:53:41.157841 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:41.157856 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:41.157958 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:41.158314 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:41.158317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:41.158628 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:41.158869 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.158877 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.159995 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.160000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.160675 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.160683 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.161368 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.161371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.161496 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.162059 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:41.162073 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:41.180201 (+ 18128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:41.182838 (+ 2637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.182842 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.182844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.183468 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.184606 (+ 1138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.184609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.184611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.184695 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.185167 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:41.185171 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:41.185262 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 10:53:41.185358 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 10:53:41.185509 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":29,"thread_start_us":117,"threads_started":1}
I20260927 10:53:41.186049 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.158610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48044 (local address 127.14.68.60:35011)
0927 10:53:41.158705 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.158710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.159224 (+ 514us) server_negotiation.cc:407] Connection header received
0927 10:53:41.159353 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.159356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.159387 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.159426 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:41.160037 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.161491 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.162532 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.162867 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.162986 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.181215 (+ 18229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:41.181221 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:41.181227 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:41.181261 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:41.183937 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.184453 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.184458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.184463 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.184545 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.184864 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.184867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.184870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.185119 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:41.185235 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.185803 (+ 568us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.185918 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":791,"server-negotiator.queue_time_us":34}
I20260927 10:53:41.186254 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.158008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40796 (local address 127.14.68.62:40605)
0927 10:53:41.158126 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.158131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.158141 (+ 10us) server_negotiation.cc:407] Connection header received
0927 10:53:41.158171 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.158173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.158209 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.158298 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:41.159014 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.159867 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.160805 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.161145 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.161240 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.180395 (+ 19155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:41.180400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:41.180403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:41.180437 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:41.182274 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.184409 (+ 2135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.184413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.184415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.184505 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.184791 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.184806 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.184808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.185038 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:41.185125 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.185959 (+ 834us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.186097 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:53:41.186371 21456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.158725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:35011 (local address 127.0.0.1:48044)
0927 10:53:41.159151 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:41.159163 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:41.159272 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:41.159534 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:41.159536 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:41.159816 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:41.159930 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.159940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.161597 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.161601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.162406 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.162418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.162986 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.162988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.163058 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.163506 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:41.163526 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:41.181043 (+ 17517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:41.184060 (+ 3017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.184065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.184069 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.184340 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.184643 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.184648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.184653 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.184759 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.185275 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:41.185279 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:41.185379 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 10:53:41.185489 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 10:53:41.185647 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":98,"threads_started":1}
I20260927 10:53:41.189486 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.157950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39728 (local address 127.14.68.61:33935)
0927 10:53:41.158075 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.158079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.158287 (+ 208us) server_negotiation.cc:407] Connection header received
0927 10:53:41.158460 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.158464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.158536 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.158592 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:41.159328 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.160206 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.161463 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.161847 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.161934 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.182478 (+ 20544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:41.182482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:41.182485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:41.182517 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:41.184269 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.187318 (+ 3049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.187322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.187325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.187424 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.187779 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.187785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.187789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.188058 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:41.188159 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.189081 (+ 922us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.189258 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:53:41.191171 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.157655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:33935 (local address 127.0.0.1:39728)
0927 10:53:41.158209 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:41.158223 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:41.158351 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:41.158809 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:41.158814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:41.159066 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:41.159184 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.159191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.160379 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.160383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.161321 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.161332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.162099 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.162102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.162196 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.163127 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:41.163139 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:41.181448 (+ 18309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:41.186943 (+ 5495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.186946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.186948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.187190 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.187535 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.187537 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.187539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.187633 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.188248 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:41.188253 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:41.188359 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 10:53:41.189739 (+ 1380us) client_negotiation.cc:240] Negotiation successful
0927 10:53:41.189910 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
Sep 27 10:53:41 dist-test-slave-tz51 krb5kdc[20668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506419, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.62@KRBTEST.COM for oryx/127.14.68.60@KRBTEST.COM
I20260927 10:53:41.199863 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.190425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48054 (local address 127.14.68.60:35011)
0927 10:53:41.190640 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.190647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.190662 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:53:41.190735 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.190740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.190806 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.190899 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:41.191379 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.192245 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.193388 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.193732 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.193844 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.196304 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:41.196308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:41.196311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:41.196337 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:41.197978 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.198515 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.198519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.198521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.198601 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.198931 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.198935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.198936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.199154 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:41.199249 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.199551 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.199661 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260927 10:53:41.199898 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.190213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:35011 (local address 127.0.0.1:48054)
0927 10:53:41.190360 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:41.190378 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:41.190469 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:41.190943 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:41.190946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:41.191139 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:41.191244 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.191251 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.192376 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.192379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.193259 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.193270 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.193854 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.193858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.193959 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.194463 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:41.194478 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:41.196152 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:41.198100 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.198105 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.198110 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.198383 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.198698 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.198702 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.198704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.198839 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.199253 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:41.199256 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:41.199356 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 10:53:41.199529 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 10:53:41.199682 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 10:53:41.201227 20713 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.68.60:35011: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fb6e814658944ac489f240027c3e09ba"
member_type: NON_VOTER
last_known_addr {
host: "127.14.68.60"
port: 35011
}
attrs {
promote: true
}
}
dest_uuid: "e37e387fd038459890cc0553983720ae"
cas_config_opid_index: 5
I20260927 10:53:41.201867 20713 consensus_queue.cc:237] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: NON_VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: true } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: NON_VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: true } }
I20260927 10:53:41.202972 21273 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fb6e814658944ac489f240027c3e09ba (127.14.68.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: NON_VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: true } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: NON_VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: true } } }
I20260927 10:53:41.204062 21405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:41.204627 21273 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.68.60:35011
I20260927 10:53:41.204816 21273 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e37e387fd038459890cc0553983720ae" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: NON_VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: true } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: NON_VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: true } } }
I20260927 10:53:41.204905 21273 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:41.205582 21405 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 1 FOLLOWER]: Refusing update from remote peer e37e387fd038459890cc0553983720ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 10:53:41.206473 21272 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: NON_VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:41.207647 21285 master.cc:537] Master@127.14.68.60:35011 shutting down...
I20260927 10:53:41.210685 21405 raft_consensus.cc:858] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: NON_VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: NON_VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: true } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: NON_VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: true } }
W20260927 10:53:41.210780 21405 raft_consensus.cc:1565] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: NON_VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: NON_VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: true } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: NON_VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: true } }
I20260927 10:53:41.212463 21405 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER cf60ff31248349aea2c39fa73946403f (127.14.68.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: NON_VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: true } } }
I20260927 10:53:41.214241 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.203850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39740 (local address 127.14.68.61:33935)
0927 10:53:41.205206 (+ 1356us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.205210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.205348 (+ 138us) server_negotiation.cc:407] Connection header received
0927 10:53:41.205518 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.205521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.205580 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.205646 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:41.206103 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.206908 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.212716 (+ 5808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.213410 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.213494 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.213645 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.214025 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.214111 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1269}
I20260927 10:53:41.214490 21460 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e37e387fd038459890cc0553983720ae" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: NON_VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: true } } }
I20260927 10:53:41.214670 21460 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 10:53:41.214732 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.203086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:33935 (local address 127.0.0.1:39740)
0927 10:53:41.205283 (+ 2197us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:41.205297 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:41.205409 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:41.205819 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:41.205822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:41.205838 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:41.205918 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.205923 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.207049 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.207052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.212558 (+ 5506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.212573 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.213745 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.213749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.213858 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.213905 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 10:53:41.214467 (+ 562us) client_negotiation.cc:240] Negotiation successful
0927 10:53:41.214585 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2138}
I20260927 10:53:41.214754 21437 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader e37e387fd038459890cc0553983720ae. Latest consensus state: current_term: 1 leader_uuid: "e37e387fd038459890cc0553983720ae" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: NON_VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: true } } }
I20260927 10:53:41.214869 21437 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 10:53:41.214960 21460 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e37e387fd038459890cc0553983720ae" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: NON_VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: true } } }
I20260927 10:53:41.215072 21460 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 10:53:41.215250 21323 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 1 LEARNER]: Refusing update from remote peer e37e387fd038459890cc0553983720ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 10:53:41.215677 21272 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: NON_VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 10:53:41.217453 21323 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fb6e814658944ac489f240027c3e09ba (127.14.68.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: NON_VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: true } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: NON_VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: true } } }
I20260927 10:53:41.219277 21352 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e37e387fd038459890cc0553983720ae. Latest consensus state: current_term: 1 leader_uuid: "e37e387fd038459890cc0553983720ae" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: NON_VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: true } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: NON_VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: true } } }
I20260927 10:53:41.219386 21352 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [sys.catalog]: This master's current role is: LEARNER
I20260927 10:53:41.219458 21352 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e37e387fd038459890cc0553983720ae" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: NON_VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: true } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: NON_VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: true } } }
I20260927 10:53:41.219523 21352 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [sys.catalog]: This master's current role is: LEARNER
I20260927 10:53:41.219957 21272 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae: attempting to promote NON_VOTER cf60ff31248349aea2c39fa73946403f to VOTER
I20260927 10:53:41.220330 21272 consensus_queue.cc:237] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: NON_VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: true } }
I20260927 10:53:41.221383 21273 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e37e387fd038459890cc0553983720ae" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: NON_VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: true } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: NON_VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: NON_VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: true } } }
I20260927 10:53:41.221504 21273 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:41.221401 21323 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 1 LEARNER]: Refusing update from remote peer e37e387fd038459890cc0553983720ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 10:53:41.221937 21273 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260927 10:53:41.222196 20700 consensus_peers.cc:487] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae -> Peer fb6e814658944ac489f240027c3e09ba (127.14.68.60:35011): Couldn't send request to peer fb6e814658944ac489f240027c3e09ba. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 10:53:41.223181 21285 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260927 10:53:41.223393 21285 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260927 10:53:41.223472 21285 tablet_replica.cc:330] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba: stopping tablet replica
I20260927 10:53:41.223541 21273 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, cf60ff31248349aea2c39fa73946403f (127.14.68.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: NON_VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: true } } }
I20260927 10:53:41.224912 21464 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e37e387fd038459890cc0553983720ae" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: NON_VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: true } } }
I20260927 10:53:41.224947 21273 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e37e387fd038459890cc0553983720ae" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: NON_VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: true } } }
I20260927 10:53:41.225020 21464 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:41.225066 21273 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:41.225821 21322 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, cf60ff31248349aea2c39fa73946403f (127.14.68.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: NON_VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: true } } }
I20260927 10:53:41.227314 21463 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e37e387fd038459890cc0553983720ae" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: NON_VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: true } } }
I20260927 10:53:41.227429 21463 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:53:41.244258 21285 master.cc:552] Master@127.14.68.60:35011 shutdown complete.
I20260927 10:53:41.244331 21285 master_runner.cc:305] Clearing existing system tablet
I20260927 10:53:41.244573 21285 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:41.244634 21285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:41.245980 21285 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 10:53:41.246923 21470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:41.247207 21285 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:41.247327 21285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 10:53:41.247361 21285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/wal
uuid: "fb6e814658944ac489f240027c3e09ba"
format_stamp: "Formatted at 2026-09-27 10:53:41 on dist-test-slave-tz51"
server_key: "cdc4c326da93fbdf3a372368155aabc4"
server_key_iv: "17926e42762bf471ab5a9b220c8d1970"
server_key_version: "encryptionkey@0"
I20260927 10:53:41.249358 21285 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:41.252204 21285 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 10:53:41.252295 21285 log.cc:1196] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 10:53:41.252568 21285 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba: Deleting consensus metadata
I20260927 10:53:41.252723 21285 master_runner.cc:315] Copying system tablet from 127.14.68.62:40605
I20260927 10:53:41.253844 21285 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba: tablet copy: Beginning tablet copy session from remote peer at address 127.14.68.62:40605
I20260927 10:53:41.267858 21476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.254436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.0.0.1:40810)
0927 10:53:41.259009 (+ 4573us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:41.259029 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:41.259205 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:41.259554 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:41.259558 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:41.259835 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:41.259956 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.259966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.261109 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.261114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.261982 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.261993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.262703 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.262707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.262813 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.263466 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:41.263482 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:41.263963 (+ 481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:41.266096 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.266100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.266102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.266356 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.266665 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.266668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.266670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.266768 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.267240 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:41.267243 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:41.267350 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 10:53:41.267455 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 10:53:41.267608 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4474,"thread_start_us":88,"threads_started":1}
I20260927 10:53:41.268405 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.254507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40810 (local address 127.14.68.62:40605)
0927 10:53:41.254635 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.254640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.259150 (+ 4510us) server_negotiation.cc:407] Connection header received
0927 10:53:41.259320 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.259324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.259387 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.259513 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:41.260141 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.260977 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.262123 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.262461 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.262563 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.264118 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:41.264122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:41.264133 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:41.264161 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:41.265963 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.266471 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.266475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.266477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.266568 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.266880 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.266884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.266885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.267135 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:41.267239 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.267560 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.267689 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:53:41.269049 20733 tablet_copy_service.cc:140] P e37e387fd038459890cc0553983720ae: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fb6e814658944ac489f240027c3e09ba ({username='oryx', principal='oryx/127.14.68.60@KRBTEST.COM'} at 127.0.0.1:40810)
I20260927 10:53:41.269161 20733 tablet_copy_service.cc:161] P e37e387fd038459890cc0553983720ae: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fb6e814658944ac489f240027c3e09ba at {username='oryx', principal='oryx/127.14.68.60@KRBTEST.COM'} at 127.0.0.1:40810: session id = fb6e814658944ac489f240027c3e09ba-00000000000000000000000000000000
I20260927 10:53:41.270346 20733 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 10:53:41.271688 21285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 10:53:41.274199 21285 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba: tablet copy: Starting download of 0 data blocks...
I20260927 10:53:41.274374 21285 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba: tablet copy: Starting download of 1 WAL segments...
I20260927 10:53:41.275959 21285 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 10:53:41.277798 20733 tablet_copy_service.cc:342] P e37e387fd038459890cc0553983720ae: Request end of tablet copy session fb6e814658944ac489f240027c3e09ba-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.68.60@KRBTEST.COM'} at 127.0.0.1:40810
I20260927 10:53:41.277876 20733 tablet_copy_service.cc:433] P e37e387fd038459890cc0553983720ae: ending tablet copy session fb6e814658944ac489f240027c3e09ba-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fb6e814658944ac489f240027c3e09ba
I20260927 10:53:41.286089 21285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:41.288051 21477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:41.288412 21480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:41.288695 21478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:41.288739 21285 server_base.cc:1034] running on GCE node
I20260927 10:53:41.289059 21285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:41.289408 21285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:41.290613 21285 hybrid_clock.cc:648] HybridClock initialized: now 1790506421290600 us; error 32 us; skew 500 ppm
Sep 27 10:53:41 dist-test-slave-tz51 krb5kdc[20668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506421, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:41.292438 21285 init.cc:377] Logged in from keytab as oryx/127.14.68.60@KRBTEST.COM (short username oryx)
I20260927 10:53:41.292965 21285 webserver.cc:460] Webserver started at http://127.14.68.60:36683/ using document root <none> and password file <none>
I20260927 10:53:41.293220 21285 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:41.293308 21285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:41.294765 21285 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:53:41.295871 21487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:41.296150 21285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:53:41.296221 21285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/wal
uuid: "fb6e814658944ac489f240027c3e09ba"
format_stamp: "Formatted at 2026-09-27 10:53:41 on dist-test-slave-tz51"
server_key: "cdc4c326da93fbdf3a372368155aabc4"
server_key_iv: "17926e42762bf471ab5a9b220c8d1970"
server_key_version: "encryptionkey@0"
I20260927 10:53:41.296294 21285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:41.326978 21285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:41.328857 21285 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260927 10:53:41.328755 21457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:41.329082 21457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:41.329171 21457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 10:53:41.332661 21038 heartbeater.cc:503] Master 127.14.68.62:40605 was elected leader, sending a full tablet report...
I20260927 10:53:41.333528 21285 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.60:35011
I20260927 10:53:41.334263 21285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/master-2/data/info.pb
I20260927 10:53:41.334401 21542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.60:35011 every 8 connection(s)
I20260927 10:53:41.335759 21543 sys_catalog.cc:263] Verifying existing consensus state
W20260927 10:53:41.335794 21457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:41.335860 21457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:41.335958 21457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:53:41.336670 21543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba: Bootstrap starting.
I20260927 10:53:41.341602 21457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:46363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.1.keytab
--principal=oryx/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:36925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=44933
--tserver_master_addrs=127.14.68.62:40605,127.14.68.61:33935,127.14.68.60:35011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:41.342708 21457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:41.343235 21543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:53:41.343526 21543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba: Bootstrap complete.
I20260927 10:53:41.343895 21457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:53:41.344120 21543 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: NON_VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: true } }
I20260927 10:53:41.344254 21543 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fb6e814658944ac489f240027c3e09ba, State: Initialized, Role: LEARNER
I20260927 10:53:41.344404 21543 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: NON_VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: true } }
I20260927 10:53:41.344808 21543 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:41.344748 21546 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: NON_VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: true } } }
I20260927 10:53:41.344901 21546 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [sys.catalog]: This master's current role is: LEARNER
I20260927 10:53:41.348275 21557 catalog_manager.cc:1259] Loaded cluster ID: a738ba3f15fa434495982601af43bfd6
I20260927 10:53:41.348335 21557 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba: loading cluster ID for follower catalog manager: success
I20260927 10:53:41.350335 21557 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba: acquiring CA information for follower catalog manager: success
I20260927 10:53:41.351552 21557 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 10:53:41.354871 21559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:41.355116 21457 server_base.cc:1034] running on GCE node
W20260927 10:53:41.355155 21562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:41.354914 21560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:41.355523 21457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:41.356213 21457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:41.357396 21457 hybrid_clock.cc:648] HybridClock initialized: now 1790506421357380 us; error 26 us; skew 500 ppm
Sep 27 10:53:41 dist-test-slave-tz51 krb5kdc[20668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506421, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:41.360255 21457 init.cc:377] Logged in from keytab as oryx/127.14.68.1@KRBTEST.COM (short username oryx)
I20260927 10:53:41.361068 21457 webserver.cc:460] Webserver started at http://127.14.68.1:44933/ using document root <none> and password file <none>
I20260927 10:53:41.361560 21457 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:41.361634 21457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:41.365207 21457 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:41.367139 21569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:41.368405 21457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:41.368495 21457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "8f96cff0ec274e9eac28349a5a876141"
format_stamp: "Formatted at 2026-09-27 10:53:40 on dist-test-slave-tz51"
server_key: "4746ab0954b2ae9d576690a134cfb83c"
server_key_iv: "4e0b2302919ffc25e86d4ed69c5d395c"
server_key_version: "encryptionkey@0"
I20260927 10:53:41.369007 21457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:41.390012 21457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:41.392896 21457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:41.393051 21457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:41.393587 21457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:41.394414 21457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:41.394461 21457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:41.394500 21457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:41.394516 21457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:41.404588 21457 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:36925
I20260927 10:53:41.404605 21682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:36925 every 8 connection(s)
I20260927 10:53:41.405663 21457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:53:41.407107 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 21457
I20260927 10:53:41.407262 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 20909
I20260927 10:53:41.414685 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:43831
--local_ip_for_outbound_sockets=127.14.68.2
--tserver_master_addrs=127.14.68.62:40605
--webserver_port=36703
--webserver_interface=127.14.68.2
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.2.keytab
--principal=oryx/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:46363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.68.62:40605,127.14.68.61:33935
--tserver_master_addrs=127.14.68.62:40605,127.14.68.61:33935,127.14.68.60:35011 with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:53:41 dist-test-slave-tz51 krb5kdc[20668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506421, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.61@KRBTEST.COM
Sep 27 10:53:41 dist-test-slave-tz51 krb5kdc[20668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506421, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.60@KRBTEST.COM
Sep 27 10:53:41 dist-test-slave-tz51 krb5kdc[20668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506421, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM
I20260927 10:53:41.423857 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.408677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:48695 (local address 127.14.68.62:40605)
0927 10:53:41.408851 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.408858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.412300 (+ 3442us) server_negotiation.cc:407] Connection header received
0927 10:53:41.412441 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.412444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.412504 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.412561 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:41.413135 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.413855 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.416939 (+ 3084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.417288 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.417388 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.420224 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:41.420231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:41.420236 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:41.420275 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:41.422204 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.422649 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.422652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.422655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.422732 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.423008 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.423011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.423013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.423212 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:41.423295 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.423589 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.423713 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 10:53:41.423982 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.408100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:55261 (local address 127.14.68.61:33935)
0927 10:53:41.408265 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.408272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.409614 (+ 1342us) server_negotiation.cc:407] Connection header received
0927 10:53:41.410536 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.410543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.410610 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.410709 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:41.412572 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.413364 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.415105 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.415442 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.415530 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.419785 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:41.419789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:41.419794 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:41.419823 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:41.421835 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.422394 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.422401 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.422405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.422501 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.422843 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.422849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.422854 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.423098 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:41.423191 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.423601 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.423765 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:53:41.424685 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.408346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.1:48695)
0927 10:53:41.412203 (+ 3857us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:41.412221 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:41.412322 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:41.412695 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:41.412698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:41.412883 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:41.412989 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.412995 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.413988 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.413991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.416796 (+ 2805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.416808 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.417575 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.417578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.417674 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.418105 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:41.418122 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:41.420015 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:41.422305 (+ 2290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.422308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.422309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.422548 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.422833 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.422835 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.422837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.422911 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.423345 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:41.423348 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:41.423422 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 10:53:41.424044 (+ 622us) client_negotiation.cc:240] Negotiation successful
0927 10:53:41.424249 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3770,"mutex_wait_us":3487,"spinlock_wait_cycles":3328,"thread_start_us":111,"threads_started":1}
I20260927 10:53:41.424926 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.408763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:35011 (local address 127.14.68.1:52811)
0927 10:53:41.410638 (+ 1875us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:41.410649 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:41.410805 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:41.411117 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:41.411120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:41.411508 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:41.412421 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.412441 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.413912 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.413918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.414968 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.414978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.415744 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.415748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.416039 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.416777 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:41.416795 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:41.419241 (+ 2446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:41.421465 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.421472 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.421488 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.421841 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.422165 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.422168 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.422170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.422338 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.422840 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:41.422846 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:41.423133 (+ 287us) client_negotiation.cc:769] Sending connection context
0927 10:53:41.423310 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 10:53:41.423648 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1820,"mutex_wait_us":234,"spinlock_wait_cycles":2176,"thread_start_us":103,"threads_started":1}
I20260927 10:53:41.425473 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.408592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:52811 (local address 127.14.68.60:35011)
0927 10:53:41.409157 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.409164 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.410738 (+ 1574us) server_negotiation.cc:407] Connection header received
0927 10:53:41.410917 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.410924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.410986 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.411083 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:41.412593 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.413777 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.415105 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.415476 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.415567 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.419433 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:41.419438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:41.419440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:41.419469 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:41.421334 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.421963 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.421969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.421972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.422060 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.422451 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.422456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.422458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.422701 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:41.422791 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.425184 (+ 2393us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.425313 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":88,"threads_started":1}
I20260927 10:53:41.425882 21684 heartbeater.cc:347] Connected to a master server at 127.14.68.62:40605
I20260927 10:53:41.425896 21687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.408346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:33935 (local address 127.14.68.1:55261)
0927 10:53:41.409435 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:41.409493 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:41.410323 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:41.410933 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:41.410940 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:41.411508 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:41.412418 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.412437 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.413527 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.413530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.414968 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.414978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.415744 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.415748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.416037 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.416645 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:41.416665 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:41.419594 (+ 2929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:41.421973 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.421978 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.421982 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.422247 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.422622 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.422627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.422631 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.422728 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.423258 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:41.423262 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:41.423382 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 10:53:41.424540 (+ 1158us) client_negotiation.cc:240] Negotiation successful
0927 10:53:41.424712 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":222,"spinlock_wait_cycles":30592,"thread_start_us":144,"threads_started":1}
I20260927 10:53:41.426141 21684 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:41.426864 21684 heartbeater.cc:511] Master 127.14.68.62:40605 requested a full tablet report, sending...
I20260927 10:53:41.426882 21685 heartbeater.cc:347] Connected to a master server at 127.14.68.61:33935
I20260927 10:53:41.427407 21685 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:41.427572 21685 heartbeater.cc:511] Master 127.14.68.61:33935 requested a full tablet report, sending...
I20260927 10:53:41.426864 21683 heartbeater.cc:347] Connected to a master server at 127.14.68.60:35011
I20260927 10:53:41.427834 21683 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:41.427970 21683 heartbeater.cc:511] Master 127.14.68.60:35011 requested a full tablet report, sending...
I20260927 10:53:41.428085 20713 ts_manager.cc:194] Re-registered known tserver with Master: 8f96cff0ec274e9eac28349a5a876141 (127.14.68.1:36925)
I20260927 10:53:41.428789 20713 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:48695
I20260927 10:53:41.429076 21507 ts_manager.cc:194] Registered new tserver with Master: 8f96cff0ec274e9eac28349a5a876141 (127.14.68.1:36925)
I20260927 10:53:41.429829 21313 ts_manager.cc:194] Registered new tserver with Master: 8f96cff0ec274e9eac28349a5a876141 (127.14.68.1:36925)
W20260927 10:53:41.534090 21691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:41.534371 21691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:41.534444 21691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:41.538566 21691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:41.538625 21691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:41.538717 21691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:53:41.543915 21691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:46363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.2.keytab
--principal=oryx/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:43831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=36703
--tserver_master_addrs=127.14.68.62:40605,127.14.68.61:33935,127.14.68.60:35011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:41.544966 21691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:41.546161 21691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:53:41.546378 21172 heartbeater.cc:503] Master 127.14.68.62:40605 was elected leader, sending a full tablet report...
W20260927 10:53:41.556037 21697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:41.556046 21699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:41.556066 21696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:41.556448 21691 server_base.cc:1034] running on GCE node
I20260927 10:53:41.556847 21691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:41.557480 21691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:41.558663 21691 hybrid_clock.cc:648] HybridClock initialized: now 1790506421558649 us; error 32 us; skew 500 ppm
Sep 27 10:53:41 dist-test-slave-tz51 krb5kdc[20668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506421, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:41.561637 21691 init.cc:377] Logged in from keytab as oryx/127.14.68.2@KRBTEST.COM (short username oryx)
I20260927 10:53:41.562454 21691 webserver.cc:460] Webserver started at http://127.14.68.2:36703/ using document root <none> and password file <none>
I20260927 10:53:41.563027 21691 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:41.563154 21691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:41.566908 21691 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:41.568929 21706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:41.570161 21691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:53:41.570315 21691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "0a59c8b46a2b48c185bfc0cd886caa51"
format_stamp: "Formatted at 2026-09-27 10:53:40 on dist-test-slave-tz51"
server_key: "891aca288ee44679ad4b33c608f56e2f"
server_key_iv: "72417871db0e865fd568f108b56e9c11"
server_key_version: "encryptionkey@0"
I20260927 10:53:41.570868 21691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:41.586769 21691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:41.590109 21691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:41.590310 21691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:41.590895 21691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:41.591805 21691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:41.591871 21691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:41.591939 21691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:41.591989 21691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:41.601573 21691 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:43831
I20260927 10:53:41.601612 21819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:43831 every 8 connection(s)
I20260927 10:53:41.602684 21691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:53:41.603585 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 21691
I20260927 10:53:41.603801 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 21043
I20260927 10:53:41.612327 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:36575
--local_ip_for_outbound_sockets=127.14.68.3
--tserver_master_addrs=127.14.68.62:40605
--webserver_port=38207
--webserver_interface=127.14.68.3
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.3.keytab
--principal=oryx/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:46363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.68.62:40605,127.14.68.61:33935
--tserver_master_addrs=127.14.68.62:40605,127.14.68.61:33935,127.14.68.60:35011 with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:53:41 dist-test-slave-tz51 krb5kdc[20668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506421, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.61@KRBTEST.COM
Sep 27 10:53:41 dist-test-slave-tz51 krb5kdc[20668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506421, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM
Sep 27 10:53:41 dist-test-slave-tz51 krb5kdc[20668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506421, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.60@KRBTEST.COM
I20260927 10:53:41.623047 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.605545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:52805 (local address 127.14.68.60:35011)
0927 10:53:41.605826 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.605833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.607979 (+ 2146us) server_negotiation.cc:407] Connection header received
0927 10:53:41.608308 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.608311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.608363 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.608441 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:41.610445 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.611275 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.613322 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.613682 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.613748 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.618230 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:41.618234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:41.618236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:41.618267 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:41.620250 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.621073 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.621078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.621080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.621188 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.621652 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.621656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.621658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.621876 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:41.621961 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.622745 (+ 784us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.622878 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260927 10:53:41.623047 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.606196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:49415 (local address 127.14.68.61:33935)
0927 10:53:41.606336 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.606340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.609286 (+ 2946us) server_negotiation.cc:407] Connection header received
0927 10:53:41.609458 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.609464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.609520 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.609578 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:41.610440 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.611301 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.613324 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.613666 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.613748 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.618230 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:41.618234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:41.618236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:41.618265 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:41.620394 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.621082 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.621086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.621088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.621188 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.621646 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.621650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.621651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.621886 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:41.621961 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.622744 (+ 783us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.622877 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:53:41.624130 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.605676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:59463 (local address 127.14.68.62:40605)
0927 10:53:41.606014 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.606017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.606926 (+ 909us) server_negotiation.cc:407] Connection header received
0927 10:53:41.608342 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.608348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.608411 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.608469 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:41.610444 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.611264 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.613341 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.613737 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.613822 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.618131 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:41.618137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:41.618142 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:41.618180 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:41.621058 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.621649 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.621655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.621659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.621759 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.622118 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.622123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.622127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.622418 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:41.622528 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.623754 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.623913 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260927 10:53:41.624466 21825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.604989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:35011 (local address 127.14.68.2:52805)
0927 10:53:41.607874 (+ 2885us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:41.607885 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:41.608133 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:41.608606 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:41.608612 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:41.609276 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:41.610285 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.610296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.611450 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.611453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.613181 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.613192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.613953 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.613957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.614321 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.615200 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:41.615215 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:41.618070 (+ 2855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:41.620397 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.620404 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.620422 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.620951 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.621300 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.621303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.621305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.621531 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.622033 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:41.622040 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:41.622430 (+ 390us) client_negotiation.cc:769] Sending connection context
0927 10:53:41.623718 (+ 1288us) client_negotiation.cc:240] Negotiation successful
0927 10:53:41.623851 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2801,"mutex_wait_us":567,"spinlock_wait_cycles":57472,"thread_start_us":141,"threads_started":1}
I20260927 10:53:41.624735 21824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.604988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.2:59463)
0927 10:53:41.606754 (+ 1766us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:41.606824 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:41.608101 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:41.608641 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:41.608647 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:41.609283 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:41.610294 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.610301 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.611450 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.611453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.613178 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.613193 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.613969 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.613971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.614299 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.615045 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:41.615076 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:41.617901 (+ 2825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:41.621207 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.621213 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.621217 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.621511 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.621892 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.621898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.621901 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.621997 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.622580 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:41.622583 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:41.622700 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 10:53:41.622862 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 10:53:41.623282 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1548,"mutex_wait_us":507,"spinlock_wait_cycles":11648,"thread_start_us":126,"threads_started":1}
I20260927 10:53:41.624938 21826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.606113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:33935 (local address 127.14.68.2:49415)
0927 10:53:41.609196 (+ 3083us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:41.609212 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:41.609354 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:41.609758 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:41.609764 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:41.609987 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:41.610269 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.610293 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.611452 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.611457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.613204 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.613211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.613953 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.613957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.614217 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.615083 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:41.615101 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:41.618080 (+ 2979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:41.620503 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.620508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.620512 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.620951 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.621300 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.621303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.621305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.621531 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.622033 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:41.622040 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:41.622430 (+ 390us) client_negotiation.cc:769] Sending connection context
0927 10:53:41.623199 (+ 769us) client_negotiation.cc:240] Negotiation successful
0927 10:53:41.623384 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2998,"mutex_wait_us":302,"spinlock_wait_cycles":15232,"thread_start_us":98,"threads_started":1}
I20260927 10:53:41.626014 21821 heartbeater.cc:347] Connected to a master server at 127.14.68.62:40605
I20260927 10:53:41.626015 21822 heartbeater.cc:347] Connected to a master server at 127.14.68.61:33935
I20260927 10:53:41.626312 21822 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:41.626858 21822 heartbeater.cc:511] Master 127.14.68.61:33935 requested a full tablet report, sending...
I20260927 10:53:41.628428 21313 ts_manager.cc:194] Registered new tserver with Master: 0a59c8b46a2b48c185bfc0cd886caa51 (127.14.68.2:43831)
I20260927 10:53:41.628729 21821 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:41.628955 21821 heartbeater.cc:511] Master 127.14.68.62:40605 requested a full tablet report, sending...
I20260927 10:53:41.629706 20713 ts_manager.cc:194] Re-registered known tserver with Master: 0a59c8b46a2b48c185bfc0cd886caa51 (127.14.68.2:43831)
I20260927 10:53:41.630528 20713 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:59463
I20260927 10:53:41.633289 21820 heartbeater.cc:347] Connected to a master server at 127.14.68.60:35011
I20260927 10:53:41.633378 21820 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:41.633605 21820 heartbeater.cc:511] Master 127.14.68.60:35011 requested a full tablet report, sending...
I20260927 10:53:41.634353 21507 ts_manager.cc:194] Registered new tserver with Master: 0a59c8b46a2b48c185bfc0cd886caa51 (127.14.68.2:43831)
I20260927 10:53:41.640138 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.636449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:35011 (local address 127.0.0.1:48058)
0927 10:53:41.636606 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:41.636625 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:41.636717 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:41.636955 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:41.636958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:41.636975 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:41.637074 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.637080 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.638172 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.638176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.638917 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.638926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.639830 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.639833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.639899 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.639918 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 10:53:41.639969 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 10:53:41.640029 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 10:53:41.640232 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.636552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48058 (local address 127.14.68.60:35011)
0927 10:53:41.636678 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.636681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.636694 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:53:41.636799 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.636801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.636848 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.636924 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:41.637221 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.638041 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.639040 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.639690 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.639831 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.640002 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.640038 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.640110 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:53:41.640700 21517 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 10:53:41.640971 21546 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader e37e387fd038459890cc0553983720ae. Latest consensus state: current_term: 1 leader_uuid: "e37e387fd038459890cc0553983720ae" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: NON_VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: true } } }
I20260927 10:53:41.641112 21546 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [sys.catalog]: This master's current role is: LEARNER
W20260927 10:53:41.734921 21827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:41.735235 21827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:41.735317 21827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:41.739490 21827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:41.739588 21827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:41.739679 21827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:53:41.745060 21827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:46363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.3.keytab
--principal=oryx/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:36575
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=38207
--tserver_master_addrs=127.14.68.62:40605,127.14.68.61:33935,127.14.68.60:35011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:41.746230 21827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:41.747267 21827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:41.757017 21832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:41.757038 21833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:41.757090 21827 server_base.cc:1034] running on GCE node
W20260927 10:53:41.757038 21835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:41.757665 21827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:41.758315 21827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:41.759536 21827 hybrid_clock.cc:648] HybridClock initialized: now 1790506421759492 us; error 53 us; skew 500 ppm
Sep 27 10:53:41 dist-test-slave-tz51 krb5kdc[20668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506421, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:41.762271 21827 init.cc:377] Logged in from keytab as oryx/127.14.68.3@KRBTEST.COM (short username oryx)
I20260927 10:53:41.763190 21827 webserver.cc:460] Webserver started at http://127.14.68.3:38207/ using document root <none> and password file <none>
I20260927 10:53:41.763820 21827 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:41.763924 21827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:41.767604 21827 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:41.769625 21842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:41.770884 21827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:41.771014 21827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "198bfa5be48e491eb86d59eab104046e"
format_stamp: "Formatted at 2026-09-27 10:53:40 on dist-test-slave-tz51"
server_key: "14bfdda7da8603d8d54a759740786c47"
server_key_iv: "fcf81306d899c9967ba9cea69a53c7c6"
server_key_version: "encryptionkey@0"
I20260927 10:53:41.771544 21827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:41.786077 21827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:41.789108 21827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:41.789263 21827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:41.789873 21827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:41.790688 21827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:41.790735 21827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:41.790774 21827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:41.790789 21827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:41.800534 21827 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:36575
I20260927 10:53:41.800609 21955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:36575 every 8 connection(s)
I20260927 10:53:41.801625 21827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:53:41.810125 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 21827
Sep 27 10:53:41 dist-test-slave-tz51 krb5kdc[20668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506421, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.61@KRBTEST.COM
Sep 27 10:53:41 dist-test-slave-tz51 krb5kdc[20668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506421, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.60@KRBTEST.COM
Sep 27 10:53:41 dist-test-slave-tz51 krb5kdc[20668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506421, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM
I20260927 10:53:41.818524 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.803468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:44779 (local address 127.14.68.61:33935)
0927 10:53:41.803652 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.803660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.804673 (+ 1013us) server_negotiation.cc:407] Connection header received
0927 10:53:41.805856 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.805860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.805919 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.805997 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:41.807894 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.808482 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.809603 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.809958 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.810075 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.814191 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:41.814197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:41.814201 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:41.814244 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:41.816615 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.817110 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.817114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.817116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.817211 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.817521 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.817526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.817528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.817735 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:41.817816 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.818190 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.818308 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 10:53:41.819572 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.804037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:33935 (local address 127.14.68.3:44779)
0927 10:53:41.804516 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:41.804571 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:41.805655 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:41.806178 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:41.806185 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:41.806821 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:41.807728 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.807741 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.808631 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.808634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.809481 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.809489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.810232 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.810236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.810435 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.811118 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:41.811142 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:41.813705 (+ 2563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:41.816751 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.816755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.816757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.817005 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.817322 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.817324 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.817327 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.817411 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.817876 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:41.817882 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:41.817975 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 10:53:41.818715 (+ 740us) client_negotiation.cc:240] Negotiation successful
0927 10:53:41.818858 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":216,"spinlock_wait_cycles":19840,"thread_start_us":136,"threads_started":1}
I20260927 10:53:41.819876 21961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.804037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.3:42605)
0927 10:53:41.804516 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:41.804571 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:41.805673 (+ 1102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:41.806179 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:41.806187 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:41.806829 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:41.807712 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.807731 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.808848 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.808851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.812171 (+ 3320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.812182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.813508 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.813512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.813619 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.814118 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:41.814132 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:41.815584 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:41.817892 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.817894 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.817896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.818129 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.818419 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.818425 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.818429 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.818510 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.818947 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:41.818950 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:41.819032 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 10:53:41.819115 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 10:53:41.819235 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":2557,"spinlock_wait_cycles":71680,"thread_start_us":148,"threads_started":1}
I20260927 10:53:41.820395 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.803923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:42605 (local address 127.14.68.62:40605)
0927 10:53:41.804066 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.804070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.804672 (+ 602us) server_negotiation.cc:407] Connection header received
0927 10:53:41.805880 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.805883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.805932 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.805984 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:41.807885 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.808700 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.813019 (+ 4319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.813369 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.813501 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.815764 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:41.815769 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:41.815775 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:41.815817 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:41.817537 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.818227 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.818230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.818232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.818316 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.818608 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.818611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.818613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.818813 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:41.818905 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.820123 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.820246 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:53:41.820762 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.805273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:45709 (local address 127.14.68.60:35011)
0927 10:53:41.805396 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.805401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.805509 (+ 108us) server_negotiation.cc:407] Connection header received
0927 10:53:41.805868 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.805872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.805925 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.805983 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:41.807885 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.808700 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.810031 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.810349 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.810453 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.814113 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:41.814116 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:41.814119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:41.814142 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:41.815849 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.816557 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.816565 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.816568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.816653 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.817126 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.817129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.817131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.817396 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:41.817498 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.820454 (+ 2956us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.820574 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":51}
I20260927 10:53:41.820900 21958 heartbeater.cc:347] Connected to a master server at 127.14.68.61:33935
I20260927 10:53:41.819571 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.805123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:35011 (local address 127.14.68.3:45709)
0927 10:53:41.805418 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:41.805430 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:41.805673 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:41.806179 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:41.806187 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:41.806824 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:41.807710 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.807731 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.808848 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.808851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.809901 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.809912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.810472 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.810476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.810596 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.811269 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:41.811283 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:41.813940 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:41.816026 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.816034 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.816052 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.816433 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.816771 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:41.816775 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:41.816778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:41.817002 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:41.817553 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:41.817563 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:41.817888 (+ 325us) client_negotiation.cc:769] Sending connection context
0927 10:53:41.818105 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 10:53:41.818465 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":204,"thread_start_us":90,"threads_started":1}
I20260927 10:53:41.821031 21957 heartbeater.cc:347] Connected to a master server at 127.14.68.62:40605
I20260927 10:53:41.821128 21957 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:41.821127 21958 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:41.821718 21956 heartbeater.cc:347] Connected to a master server at 127.14.68.60:35011
I20260927 10:53:41.821748 21957 heartbeater.cc:511] Master 127.14.68.62:40605 requested a full tablet report, sending...
I20260927 10:53:41.821801 21956 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:41.821750 21958 heartbeater.cc:511] Master 127.14.68.61:33935 requested a full tablet report, sending...
I20260927 10:53:41.821947 21956 heartbeater.cc:511] Master 127.14.68.60:35011 requested a full tablet report, sending...
I20260927 10:53:41.822751 20713 ts_manager.cc:194] Re-registered known tserver with Master: 198bfa5be48e491eb86d59eab104046e (127.14.68.3:36575)
I20260927 10:53:41.822820 21507 ts_manager.cc:194] Registered new tserver with Master: 198bfa5be48e491eb86d59eab104046e (127.14.68.3:36575)
I20260927 10:53:41.822824 21313 ts_manager.cc:194] Registered new tserver with Master: 198bfa5be48e491eb86d59eab104046e (127.14.68.3:36575)
I20260927 10:53:41.823447 20713 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:42605
I20260927 10:53:41.825765 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.812273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48060 (local address 127.14.68.60:35011)
0927 10:53:41.813367 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.813379 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.813391 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:53:41.813425 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.813427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.813472 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.813553 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:41.814619 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.815494 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.816806 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.817182 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.820513 (+ 3331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.820618 (+ 105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:41.820621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:41.820634 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:41.820651 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:41.823993 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.824470 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.824475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.824476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.824584 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.824876 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.824878 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.824879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.825038 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:41.825187 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.825448 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.825560 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3215,"server-negotiator.queue_time_us":1023,"thread_start_us":75,"threads_started":1}
I20260927 10:53:41.829036 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.812180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40816 (local address 127.14.68.62:40605)
0927 10:53:41.817728 (+ 5548us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.817732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.817746 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:53:41.817783 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.817789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.817838 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.817920 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:41.819653 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.820389 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.823758 (+ 3369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.824034 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.824154 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.825782 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:41.825786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:41.825789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:41.825815 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:41.827385 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.827890 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.827893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.827895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.827983 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.828231 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.828234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.828235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.828429 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:41.828558 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.828772 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.828869 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5476,"thread_start_us":76,"threads_started":1}
I20260927 10:53:41.832301 20713 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 10:53:41.836253 21972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.812657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39742 (local address 127.14.68.61:33935)
0927 10:53:41.812928 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.812935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.825781 (+ 12846us) server_negotiation.cc:407] Connection header received
0927 10:53:41.825934 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.825938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.825991 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.826105 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:41.826973 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.827522 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.828579 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.828938 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.829036 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.830561 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:41.830566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:41.830570 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:41.830599 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:41.832375 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.832857 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.832861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.832863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.832988 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:41.833346 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:41.833349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:41.833351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:41.833600 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:41.833734 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.834144 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.834329 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":100,"threads_started":1}
W20260927 10:53:41.836719 20713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:41.839298 21465 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e37e387fd038459890cc0553983720ae" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: NON_VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: true } } }
I20260927 10:53:41.839426 21465 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:41.841454 21975 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae: attempting to promote NON_VOTER fb6e814658944ac489f240027c3e09ba to VOTER
I20260927 10:53:41.841854 21975 consensus_queue.cc:237] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: false } }
I20260927 10:53:41.842991 21322 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 1 FOLLOWER]: Refusing update from remote peer e37e387fd038459890cc0553983720ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 10:53:41.843094 21517 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 1 LEARNER]: Refusing update from remote peer e37e387fd038459890cc0553983720ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 10:53:41.843443 21974 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 10:53:41.843592 21465 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 10:53:41.845341 21975 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fb6e814658944ac489f240027c3e09ba (127.14.68.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: false } } }
I20260927 10:53:41.845717 21517 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fb6e814658944ac489f240027c3e09ba (127.14.68.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: false } } }
I20260927 10:53:41.846836 21975 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e37e387fd038459890cc0553983720ae" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: false } } }
I20260927 10:53:41.846809 21974 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e37e387fd038459890cc0553983720ae" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: false } } }
I20260927 10:53:41.846921 21975 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:41.846933 21974 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:41.847280 21546 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e37e387fd038459890cc0553983720ae" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: false } } }
I20260927 10:53:41.847397 21546 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:53:41.847407 21465 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e37e387fd038459890cc0553983720ae" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: false } } }
I20260927 10:53:41.847517 21465 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:41.848066 21322 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fb6e814658944ac489f240027c3e09ba (127.14.68.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: false } } }
I20260927 10:53:41.850006 21984 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e37e387fd038459890cc0553983720ae" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: false } } }
I20260927 10:53:41.850132 21984 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:53:41.859869 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.854543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:36575 (local address 127.0.0.1:42564)
0927 10:53:41.854698 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:41.854715 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:41.854794 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:41.855865 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:41.855867 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:41.855884 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:41.855987 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.855993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.857184 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.857191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.858263 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.858273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.859450 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.859453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.859540 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.859564 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 10:53:41.859649 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 10:53:41.859742 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 10:53:41.860049 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.854703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42564 (local address 127.14.68.3:36575)
0927 10:53:41.855329 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.855346 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.855373 (+ 27us) server_negotiation.cc:407] Connection header received
0927 10:53:41.855458 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.855467 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.855734 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.855905 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:41.856080 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.857059 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.858387 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.859304 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.859426 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.859571 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.859684 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.859800 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":120,"threads_started":1}
I20260927 10:53:41.860991 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.854952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:43831 (local address 127.0.0.1:56576)
0927 10:53:41.855174 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:41.855190 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:41.855280 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:41.856577 (+ 1297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:41.856580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:41.856598 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:41.856701 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.856708 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.857881 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.857884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.859025 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.859034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.860602 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.860606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.860700 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.860720 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:53:41.860787 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 10:53:41.860854 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":21,"thread_start_us":109,"threads_started":1}
I20260927 10:53:41.861266 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.855127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:36925 (local address 127.0.0.1:41156)
0927 10:53:41.855573 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:41.855586 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:41.855664 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:41.857371 (+ 1707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:41.857373 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:41.857390 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:41.857466 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.857471 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.858717 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.858720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.859962 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:41.859976 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.860851 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.860855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:41.860937 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.860959 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:53:41.861048 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 10:53:41.861132 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260927 10:53:41.861433 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.855390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56576 (local address 127.14.68.2:43831)
0927 10:53:41.856136 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.856149 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.856175 (+ 26us) server_negotiation.cc:407] Connection header received
0927 10:53:41.856253 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.856257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.856427 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.856595 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:41.856834 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.857765 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.859184 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.860321 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.860413 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.860536 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.861206 (+ 670us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.861297 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":122,"threads_started":1}
I20260927 10:53:41.861759 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:41.855555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41156 (local address 127.14.68.1:36925)
0927 10:53:41.856878 (+ 1323us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:41.856887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:41.856903 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:53:41.856957 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:41.856961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:41.857081 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:41.857240 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:41.857593 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.858584 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.860111 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:41.860707 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:41.860848 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:41.860974 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:41.861237 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 10:53:41.861592 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1220,"thread_start_us":105,"threads_started":1}
I20260927 10:53:41.863253 21890 tablet_service.cc:1467] Processing CreateTablet for tablet 9f252bdf024a489a98ccfe1db1aaab7d (DEFAULT_TABLE table=test-table [id=5d2f185d0922489db8c53adb51f1a86c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:41.864434 21890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f252bdf024a489a98ccfe1db1aaab7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:41.864092 21754 tablet_service.cc:1467] Processing CreateTablet for tablet 9f252bdf024a489a98ccfe1db1aaab7d (DEFAULT_TABLE table=test-table [id=5d2f185d0922489db8c53adb51f1a86c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:41.864097 21617 tablet_service.cc:1467] Processing CreateTablet for tablet 9f252bdf024a489a98ccfe1db1aaab7d (DEFAULT_TABLE table=test-table [id=5d2f185d0922489db8c53adb51f1a86c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:41.865018 21617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f252bdf024a489a98ccfe1db1aaab7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:41.865036 21754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f252bdf024a489a98ccfe1db1aaab7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:41.871587 21990 tablet_bootstrap.cc:492] T 9f252bdf024a489a98ccfe1db1aaab7d P 8f96cff0ec274e9eac28349a5a876141: Bootstrap starting.
I20260927 10:53:41.873032 21991 tablet_bootstrap.cc:492] T 9f252bdf024a489a98ccfe1db1aaab7d P 0a59c8b46a2b48c185bfc0cd886caa51: Bootstrap starting.
I20260927 10:53:41.874506 21990 tablet_bootstrap.cc:654] T 9f252bdf024a489a98ccfe1db1aaab7d P 8f96cff0ec274e9eac28349a5a876141: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:41.874610 21992 tablet_bootstrap.cc:492] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e: Bootstrap starting.
I20260927 10:53:41.875345 21990 log.cc:824] T 9f252bdf024a489a98ccfe1db1aaab7d P 8f96cff0ec274e9eac28349a5a876141: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:41.876549 21991 tablet_bootstrap.cc:654] T 9f252bdf024a489a98ccfe1db1aaab7d P 0a59c8b46a2b48c185bfc0cd886caa51: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:41.876684 21992 tablet_bootstrap.cc:654] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:41.877331 21992 log.cc:824] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:41.877367 21991 log.cc:824] T 9f252bdf024a489a98ccfe1db1aaab7d P 0a59c8b46a2b48c185bfc0cd886caa51: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:41.877560 21990 tablet_bootstrap.cc:492] T 9f252bdf024a489a98ccfe1db1aaab7d P 8f96cff0ec274e9eac28349a5a876141: No bootstrap required, opened a new log
I20260927 10:53:41.877758 21990 ts_tablet_manager.cc:1397] T 9f252bdf024a489a98ccfe1db1aaab7d P 8f96cff0ec274e9eac28349a5a876141: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:53:41.879278 21992 tablet_bootstrap.cc:492] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e: No bootstrap required, opened a new log
I20260927 10:53:41.879451 21992 ts_tablet_manager.cc:1397] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:53:41.879863 21991 tablet_bootstrap.cc:492] T 9f252bdf024a489a98ccfe1db1aaab7d P 0a59c8b46a2b48c185bfc0cd886caa51: No bootstrap required, opened a new log
I20260927 10:53:41.880026 21991 ts_tablet_manager.cc:1397] T 9f252bdf024a489a98ccfe1db1aaab7d P 0a59c8b46a2b48c185bfc0cd886caa51: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 10:53:41.881809 21990 raft_consensus.cc:348] T 9f252bdf024a489a98ccfe1db1aaab7d P 8f96cff0ec274e9eac28349a5a876141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198bfa5be48e491eb86d59eab104046e" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36575 } } peers { permanent_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43831 } } peers { permanent_uuid: "8f96cff0ec274e9eac28349a5a876141" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36925 } }
I20260927 10:53:41.882050 21990 raft_consensus.cc:374] T 9f252bdf024a489a98ccfe1db1aaab7d P 8f96cff0ec274e9eac28349a5a876141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:41.882110 21990 raft_consensus.cc:729] T 9f252bdf024a489a98ccfe1db1aaab7d P 8f96cff0ec274e9eac28349a5a876141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f96cff0ec274e9eac28349a5a876141, State: Initialized, Role: FOLLOWER
I20260927 10:53:41.882568 21990 consensus_queue.cc:260] T 9f252bdf024a489a98ccfe1db1aaab7d P 8f96cff0ec274e9eac28349a5a876141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198bfa5be48e491eb86d59eab104046e" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36575 } } peers { permanent_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43831 } } peers { permanent_uuid: "8f96cff0ec274e9eac28349a5a876141" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36925 } }
I20260927 10:53:41.883265 21684 heartbeater.cc:503] Master 127.14.68.62:40605 was elected leader, sending a full tablet report...
I20260927 10:53:41.884472 21991 raft_consensus.cc:348] T 9f252bdf024a489a98ccfe1db1aaab7d P 0a59c8b46a2b48c185bfc0cd886caa51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198bfa5be48e491eb86d59eab104046e" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36575 } } peers { permanent_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43831 } } peers { permanent_uuid: "8f96cff0ec274e9eac28349a5a876141" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36925 } }
I20260927 10:53:41.884732 21991 raft_consensus.cc:374] T 9f252bdf024a489a98ccfe1db1aaab7d P 0a59c8b46a2b48c185bfc0cd886caa51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:41.884796 21991 raft_consensus.cc:729] T 9f252bdf024a489a98ccfe1db1aaab7d P 0a59c8b46a2b48c185bfc0cd886caa51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a59c8b46a2b48c185bfc0cd886caa51, State: Initialized, Role: FOLLOWER
I20260927 10:53:41.885232 21991 consensus_queue.cc:260] T 9f252bdf024a489a98ccfe1db1aaab7d P 0a59c8b46a2b48c185bfc0cd886caa51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198bfa5be48e491eb86d59eab104046e" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36575 } } peers { permanent_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43831 } } peers { permanent_uuid: "8f96cff0ec274e9eac28349a5a876141" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36925 } }
I20260927 10:53:41.886070 21990 ts_tablet_manager.cc:1428] T 9f252bdf024a489a98ccfe1db1aaab7d P 8f96cff0ec274e9eac28349a5a876141: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 10:53:41.886232 21991 ts_tablet_manager.cc:1428] T 9f252bdf024a489a98ccfe1db1aaab7d P 0a59c8b46a2b48c185bfc0cd886caa51: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:53:41.887266 21821 heartbeater.cc:503] Master 127.14.68.62:40605 was elected leader, sending a full tablet report...
I20260927 10:53:41.888273 21992 raft_consensus.cc:348] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198bfa5be48e491eb86d59eab104046e" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36575 } } peers { permanent_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43831 } } peers { permanent_uuid: "8f96cff0ec274e9eac28349a5a876141" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36925 } }
I20260927 10:53:41.888501 21992 raft_consensus.cc:374] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:41.888561 21992 raft_consensus.cc:729] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 198bfa5be48e491eb86d59eab104046e, State: Initialized, Role: FOLLOWER
I20260927 10:53:41.888983 21992 consensus_queue.cc:260] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198bfa5be48e491eb86d59eab104046e" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36575 } } peers { permanent_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43831 } } peers { permanent_uuid: "8f96cff0ec274e9eac28349a5a876141" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36925 } }
I20260927 10:53:41.889922 21992 ts_tablet_manager.cc:1428] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260927 10:53:41.890151 21957 heartbeater.cc:503] Master 127.14.68.62:40605 was elected leader, sending a full tablet report...
W20260927 10:53:41.907240 21686 tablet.cc:2369] T 9f252bdf024a489a98ccfe1db1aaab7d P 8f96cff0ec274e9eac28349a5a876141: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:53:42.053512 21959 tablet.cc:2369] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:53:42.104228 21823 tablet.cc:2369] T 9f252bdf024a489a98ccfe1db1aaab7d P 0a59c8b46a2b48c185bfc0cd886caa51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:53:42.255411 21998 raft_consensus.cc:482] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:42.255653 21998 raft_consensus.cc:504] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198bfa5be48e491eb86d59eab104046e" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36575 } } peers { permanent_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43831 } } peers { permanent_uuid: "8f96cff0ec274e9eac28349a5a876141" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36925 } }
I20260927 10:53:42.257048 21998 leader_election.cc:288] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a59c8b46a2b48c185bfc0cd886caa51 (127.14.68.2:43831), 8f96cff0ec274e9eac28349a5a876141 (127.14.68.1:36925)
I20260927 10:53:42.260929 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:42.257409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:42705 (local address 127.14.68.1:36925)
0927 10:53:42.257531 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:42.257535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:42.257577 (+ 42us) server_negotiation.cc:407] Connection header received
0927 10:53:42.257747 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:42.257750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:42.257792 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:42.257875 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:42.258201 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.258708 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.259891 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.260355 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.260467 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:42.260536 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:42.260730 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 10:53:42.260798 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:53:42.260937 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:42.257333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:36925 (local address 127.14.68.3:42705)
0927 10:53:42.257513 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:42.257527 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:42.257635 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:42.257885 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:42.257887 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:42.257902 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:42.258059 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:42.258066 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.258822 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.258826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:42.259773 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:42.259783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.260471 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.260474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:42.260598 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:42.260627 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 10:53:42.260716 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 10:53:42.260793 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":31}
I20260927 10:53:42.260949 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:42.257455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:51903 (local address 127.14.68.2:43831)
0927 10:53:42.257599 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:42.257609 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:42.257622 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:53:42.257671 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:42.257673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:42.257712 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:42.257794 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:42.258104 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.258623 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.259894 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.260360 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.260478 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:42.260543 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:42.260758 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 10:53:42.260829 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:53:42.260949 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:42.257242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:43831 (local address 127.14.68.3:51903)
0927 10:53:42.257390 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:42.257406 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:42.257504 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:42.257819 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:42.257822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:42.257837 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:42.257981 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:42.257989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.258740 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.258746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:42.259776 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:42.259785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.260478 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.260481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:42.260631 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:42.260653 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:53:42.260740 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 10:53:42.260817 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":47}
I20260927 10:53:42.261624 21774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f252bdf024a489a98ccfe1db1aaab7d" candidate_uuid: "198bfa5be48e491eb86d59eab104046e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" is_pre_election: true
I20260927 10:53:42.261615 21637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f252bdf024a489a98ccfe1db1aaab7d" candidate_uuid: "198bfa5be48e491eb86d59eab104046e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f96cff0ec274e9eac28349a5a876141" is_pre_election: true
I20260927 10:53:42.261922 21637 raft_consensus.cc:2454] T 9f252bdf024a489a98ccfe1db1aaab7d P 8f96cff0ec274e9eac28349a5a876141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 198bfa5be48e491eb86d59eab104046e in term 0.
I20260927 10:53:42.261922 21774 raft_consensus.cc:2454] T 9f252bdf024a489a98ccfe1db1aaab7d P 0a59c8b46a2b48c185bfc0cd886caa51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 198bfa5be48e491eb86d59eab104046e in term 0.
I20260927 10:53:42.262422 21843 leader_election.cc:302] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0a59c8b46a2b48c185bfc0cd886caa51, 198bfa5be48e491eb86d59eab104046e, 8f96cff0ec274e9eac28349a5a876141; no voters:
I20260927 10:53:42.262728 21998 raft_consensus.cc:2781] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:42.262830 21998 raft_consensus.cc:482] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:42.262894 21998 raft_consensus.cc:3037] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:42.263952 21998 raft_consensus.cc:504] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198bfa5be48e491eb86d59eab104046e" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36575 } } peers { permanent_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43831 } } peers { permanent_uuid: "8f96cff0ec274e9eac28349a5a876141" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36925 } }
I20260927 10:53:42.264309 21998 leader_election.cc:288] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e [CANDIDATE]: Term 1 election: Requested vote from peers 0a59c8b46a2b48c185bfc0cd886caa51 (127.14.68.2:43831), 8f96cff0ec274e9eac28349a5a876141 (127.14.68.1:36925)
I20260927 10:53:42.264655 21774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f252bdf024a489a98ccfe1db1aaab7d" candidate_uuid: "198bfa5be48e491eb86d59eab104046e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a59c8b46a2b48c185bfc0cd886caa51"
I20260927 10:53:42.264667 21637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f252bdf024a489a98ccfe1db1aaab7d" candidate_uuid: "198bfa5be48e491eb86d59eab104046e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f96cff0ec274e9eac28349a5a876141"
I20260927 10:53:42.264770 21774 raft_consensus.cc:3037] T 9f252bdf024a489a98ccfe1db1aaab7d P 0a59c8b46a2b48c185bfc0cd886caa51 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:42.264810 21637 raft_consensus.cc:3037] T 9f252bdf024a489a98ccfe1db1aaab7d P 8f96cff0ec274e9eac28349a5a876141 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:42.265754 21774 raft_consensus.cc:2454] T 9f252bdf024a489a98ccfe1db1aaab7d P 0a59c8b46a2b48c185bfc0cd886caa51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 198bfa5be48e491eb86d59eab104046e in term 1.
I20260927 10:53:42.265852 21637 raft_consensus.cc:2454] T 9f252bdf024a489a98ccfe1db1aaab7d P 8f96cff0ec274e9eac28349a5a876141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 198bfa5be48e491eb86d59eab104046e in term 1.
I20260927 10:53:42.266022 21843 leader_election.cc:302] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a59c8b46a2b48c185bfc0cd886caa51, 198bfa5be48e491eb86d59eab104046e; no voters:
I20260927 10:53:42.266247 21998 raft_consensus.cc:2781] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:42.266469 21998 raft_consensus.cc:686] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e [term 1 LEADER]: Becoming Leader. State: Replica: 198bfa5be48e491eb86d59eab104046e, State: Running, Role: LEADER
I20260927 10:53:42.267009 21998 consensus_queue.cc:237] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198bfa5be48e491eb86d59eab104046e" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36575 } } peers { permanent_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43831 } } peers { permanent_uuid: "8f96cff0ec274e9eac28349a5a876141" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36925 } }
I20260927 10:53:42.270475 20713 catalog_manager.cc:5495] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e reported cstate change: term changed from 0 to 1, leader changed from <none> to 198bfa5be48e491eb86d59eab104046e (127.14.68.3). New cstate: current_term: 1 leader_uuid: "198bfa5be48e491eb86d59eab104046e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198bfa5be48e491eb86d59eab104046e" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f96cff0ec274e9eac28349a5a876141" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36925 } health_report { overall_health: UNKNOWN } } }
I20260927 10:53:42.313405 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:42.309922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.14.68.3:36575)
0927 10:53:42.310055 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:42.310059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:42.310072 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:53:42.310147 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:42.310149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:42.310258 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:42.310372 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:42.310573 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.311109 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.312089 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.312309 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.312388 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:42.312634 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:42.313051 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:42.313141 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:42.313237 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 10:53:42.313287 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:53:42.319033 21637 raft_consensus.cc:1261] T 9f252bdf024a489a98ccfe1db1aaab7d P 8f96cff0ec274e9eac28349a5a876141 [term 1 FOLLOWER]: Refusing update from remote peer 198bfa5be48e491eb86d59eab104046e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:42.319056 21774 raft_consensus.cc:1261] T 9f252bdf024a489a98ccfe1db1aaab7d P 0a59c8b46a2b48c185bfc0cd886caa51 [term 1 FOLLOWER]: Refusing update from remote peer 198bfa5be48e491eb86d59eab104046e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:42.319813 21999 consensus_queue.cc:1035] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:42.319983 21998 consensus_queue.cc:1035] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f96cff0ec274e9eac28349a5a876141" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:42.327546 22002 mvcc.cc:205] Tried to move back new op lower bound from 7333914305811734528 to 7333914305608380416. Current Snapshot: MvccSnapshot[applied={T|T < 7333914305811734528 or (T in {7333914305811734528})}]
I20260927 10:53:42.336922 20712 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:42.337082 20712 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:42.343189 20712 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e37e387fd038459890cc0553983720ae: Sending DeleteTablet for 3 replicas of tablet 9f252bdf024a489a98ccfe1db1aaab7d
I20260927 10:53:42.343947 21754 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f252bdf024a489a98ccfe1db1aaab7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:42 UTC) from {username='oryx', principal='oryx/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:56576
I20260927 10:53:42.344132 21890 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f252bdf024a489a98ccfe1db1aaab7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:42 UTC) from {username='oryx', principal='oryx/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:42564
I20260927 10:53:42.344449 22012 tablet_replica.cc:330] T 9f252bdf024a489a98ccfe1db1aaab7d P 0a59c8b46a2b48c185bfc0cd886caa51: stopping tablet replica
I20260927 10:53:42.344558 21889 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f252bdf024a489a98ccfe1db1aaab7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:42 UTC) from {username='oryx', principal='oryx/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:42564
I20260927 10:53:42.344913 21617 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f252bdf024a489a98ccfe1db1aaab7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:42 UTC) from {username='oryx', principal='oryx/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:41156
I20260927 10:53:42.345397 22014 tablet_replica.cc:330] T 9f252bdf024a489a98ccfe1db1aaab7d P 8f96cff0ec274e9eac28349a5a876141: stopping tablet replica
I20260927 10:53:42.345573 21617 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f252bdf024a489a98ccfe1db1aaab7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:42 UTC) from {username='oryx', principal='oryx/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:41156
I20260927 10:53:42.345890 22014 raft_consensus.cc:2229] T 9f252bdf024a489a98ccfe1db1aaab7d P 8f96cff0ec274e9eac28349a5a876141 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:42.346288 22014 raft_consensus.cc:2258] T 9f252bdf024a489a98ccfe1db1aaab7d P 8f96cff0ec274e9eac28349a5a876141 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:42.347579 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 20681
I20260927 10:53:42.356226 22013 tablet_replica.cc:330] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e: stopping tablet replica
I20260927 10:53:42.356598 22013 raft_consensus.cc:2229] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:53:42.357012 22013 raft_consensus.cc:2258] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:42.358863 22013 ts_tablet_manager.cc:1905] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:42.359138 22012 raft_consensus.cc:2229] T 9f252bdf024a489a98ccfe1db1aaab7d P 0a59c8b46a2b48c185bfc0cd886caa51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:42.359516 22012 raft_consensus.cc:2258] T 9f252bdf024a489a98ccfe1db1aaab7d P 0a59c8b46a2b48c185bfc0cd886caa51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:42.361142 22012 ts_tablet_manager.cc:1905] T 9f252bdf024a489a98ccfe1db1aaab7d P 0a59c8b46a2b48c185bfc0cd886caa51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:42.361908 22013 ts_tablet_manager.cc:1918] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:42.362015 22013 log.cc:1196] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/9f252bdf024a489a98ccfe1db1aaab7d
I20260927 10:53:42.362375 22013 ts_tablet_manager.cc:1939] T 9f252bdf024a489a98ccfe1db1aaab7d P 198bfa5be48e491eb86d59eab104046e: Deleting consensus metadata
W20260927 10:53:42.363543 21844 connection.cc:441] server connection from 127.0.0.1:42564 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42564 (request call id 1) could send its response
I20260927 10:53:42.365070 22012 ts_tablet_manager.cc:1918] T 9f252bdf024a489a98ccfe1db1aaab7d P 0a59c8b46a2b48c185bfc0cd886caa51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:42.365266 22012 log.cc:1196] T 9f252bdf024a489a98ccfe1db1aaab7d P 0a59c8b46a2b48c185bfc0cd886caa51: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/9f252bdf024a489a98ccfe1db1aaab7d
I20260927 10:53:42.365708 22012 ts_tablet_manager.cc:1939] T 9f252bdf024a489a98ccfe1db1aaab7d P 0a59c8b46a2b48c185bfc0cd886caa51: Deleting consensus metadata
W20260927 10:53:42.366917 21709 connection.cc:441] server connection from 127.0.0.1:56576 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56576 (request call id 1) could send its response
I20260927 10:53:42.368577 22016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:42.346974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38338 (local address 127.14.68.61:33935)
0927 10:53:42.357409 (+ 10435us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:42.357414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:42.358878 (+ 1464us) server_negotiation.cc:407] Connection header received
0927 10:53:42.358935 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:42.358938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:42.358993 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:42.359076 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:42.360093 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.360888 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.362119 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.362503 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.362607 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:42.364595 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:42.364598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:42.364601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:42.364630 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:42.366448 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:42.366944 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:42.366947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:42.366949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:42.367034 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:42.367310 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:42.367313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:42.367314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:42.367527 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:42.367658 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:42.368259 (+ 601us) server_negotiation.cc:299] Negotiation successful
0927 10:53:42.368409 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10337,"thread_start_us":73,"threads_started":1}
W20260927 10:53:42.370920 21844 connection.cc:441] server connection from 127.0.0.1:42564 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42564 (request call id 2) could send its response
I20260927 10:53:42.378947 22014 ts_tablet_manager.cc:1905] T 9f252bdf024a489a98ccfe1db1aaab7d P 8f96cff0ec274e9eac28349a5a876141: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:42.380934 22016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:42.364308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38342 (local address 127.14.68.61:33935)
0927 10:53:42.368739 (+ 4431us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:42.368743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:42.371120 (+ 2377us) server_negotiation.cc:407] Connection header received
0927 10:53:42.371216 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:42.371219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:42.371261 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:42.371340 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:42.372494 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.373305 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.374561 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.374940 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.375045 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:42.377104 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:42.377107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:42.377110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:42.377137 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:42.378891 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:42.379370 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:42.379374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:42.379375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:42.379462 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:42.379822 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:42.379826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:42.379828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:42.380034 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:42.380157 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:42.380594 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 10:53:42.380736 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4375,"thread_start_us":70,"threads_started":1}
I20260927 10:53:42.382103 22014 ts_tablet_manager.cc:1918] T 9f252bdf024a489a98ccfe1db1aaab7d P 8f96cff0ec274e9eac28349a5a876141: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:42.382202 22014 log.cc:1196] T 9f252bdf024a489a98ccfe1db1aaab7d P 8f96cff0ec274e9eac28349a5a876141: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-0/wal/wals/9f252bdf024a489a98ccfe1db1aaab7d
I20260927 10:53:42.382472 22014 ts_tablet_manager.cc:1939] T 9f252bdf024a489a98ccfe1db1aaab7d P 8f96cff0ec274e9eac28349a5a876141: Deleting consensus metadata
I20260927 10:53:42.382515 22022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:42.364011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55624 (local address 127.14.68.60:35011)
0927 10:53:42.372267 (+ 8256us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:42.372272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:42.372285 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:53:42.372322 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:42.372324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:42.372365 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:42.372445 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:42.373315 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.374181 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.376492 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.376865 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.376959 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:42.378469 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:42.378473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:42.378476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:42.378501 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:42.380237 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:42.380714 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:42.380717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:42.380719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:42.380801 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:42.381236 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:42.381239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:42.381241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:42.381446 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:42.381566 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:42.382196 (+ 630us) server_negotiation.cc:299] Negotiation successful
0927 10:53:42.382348 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8180,"thread_start_us":93,"threads_started":1}
W20260927 10:53:42.383525 21571 connection.cc:441] server connection from 127.0.0.1:41156 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41156 (request call id 1) could send its response
W20260927 10:53:42.383808 21571 connection.cc:441] server connection from 127.0.0.1:41156 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41156 (request call id 2) could send its response
I20260927 10:53:42.385777 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:42.347149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55608 (local address 127.14.68.60:35011)
0927 10:53:42.347374 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:42.347380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:42.368936 (+ 21556us) server_negotiation.cc:407] Connection header received
0927 10:53:42.369106 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:42.369109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:42.369174 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:42.369232 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:42.370098 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.370872 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.372320 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:42.372670 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:42.372747 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:42.374585 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:42.374589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:42.374592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:42.374617 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:42.376950 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:42.384617 (+ 7667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:42.384620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:42.384621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:42.384702 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:42.384982 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:42.384984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:42.384986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:42.385156 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:42.385255 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:42.385486 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 10:53:42.385588 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":62,"threads_started":1}
I20260927 10:53:43.332815 22027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.332396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.2:45391)
0927 10:53:43.332647 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:43.332725 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":91,"threads_started":1}
W20260927 10:53:43.333140 21821 heartbeater.cc:644] Failed to heartbeat to 127.14.68.62:40605 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
I20260927 10:53:43.344857 22028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.344440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.1:50093)
0927 10:53:43.344679 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:43.344753 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":78,"threads_started":1}
W20260927 10:53:43.345268 21684 heartbeater.cc:644] Failed to heartbeat to 127.14.68.62:40605 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
I20260927 10:53:43.346767 22029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.346378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.3:38355)
0927 10:53:43.346598 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:43.346685 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":90,"threads_started":1}
W20260927 10:53:43.347123 21957 heartbeater.cc:644] Failed to heartbeat to 127.14.68.62:40605 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
I20260927 10:53:43.842374 22030 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e37e387fd038459890cc0553983720ae)
I20260927 10:53:43.842753 22031 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e37e387fd038459890cc0553983720ae)
I20260927 10:53:43.842854 22030 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: false } }
I20260927 10:53:43.842983 22031 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: false } }
I20260927 10:53:43.844010 22030 leader_election.cc:288] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e37e387fd038459890cc0553983720ae (127.14.68.62:40605), cf60ff31248349aea2c39fa73946403f (127.14.68.61:33935)
I20260927 10:53:43.844336 22031 leader_election.cc:288] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e37e387fd038459890cc0553983720ae (127.14.68.62:40605), fb6e814658944ac489f240027c3e09ba (127.14.68.60:35011)
I20260927 10:53:43.844610 22034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.844136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.0.0.1:47258)
0927 10:53:43.844387 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:43.844473 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":108,"threads_started":1}
W20260927 10:53:43.844789 21299 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
I20260927 10:53:43.844907 22035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.844126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.0.0.1:47256)
0927 10:53:43.844753 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:43.844832 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":85,"threads_started":1}
W20260927 10:53:43.845260 21490 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
I20260927 10:53:43.846362 22033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.846140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.0.0.1:47262)
0927 10:53:43.846224 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:43.846282 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 10:53:43.846553 21299 leader_election.cc:334] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e37e387fd038459890cc0553983720ae (127.14.68.62:40605): Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
I20260927 10:53:43.846711 22032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.846431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.0.0.1:47278)
0927 10:53:43.846574 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:43.846634 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260927 10:53:43.846835 21490 leader_election.cc:334] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e37e387fd038459890cc0553983720ae (127.14.68.62:40605): Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
I20260927 10:53:43.849500 22035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.844516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:33935 (local address 127.0.0.1:38358)
0927 10:53:43.845063 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:43.845077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:43.845187 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:43.845653 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:43.845656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:43.845700 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:43.845802 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:43.845807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.846939 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.846942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:43.848276 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:43.848287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.849177 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.849180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:43.849260 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:43.849280 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:53:43.849347 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 10:53:43.849406 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":102,"threads_started":1}
I20260927 10:53:43.849519 22036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.844600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38358 (local address 127.14.68.61:33935)
0927 10:53:43.845394 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:43.845398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:43.845411 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:53:43.845459 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:43.845462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:43.845520 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:43.845950 (+ 430us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:43.846034 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.846807 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.848432 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.849056 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.849167 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:43.849232 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:43.849365 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 10:53:43.849424 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":53,"threads_started":1}
I20260927 10:53:43.850085 22034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.844160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:35011 (local address 127.0.0.1:55632)
0927 10:53:43.845239 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:43.845255 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:43.845395 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:43.845661 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:43.845664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:43.845681 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:43.845778 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:43.845784 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.846941 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.846944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:43.848276 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:43.848288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.849704 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.849708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:43.849801 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:43.849823 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:53:43.849895 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 10:53:43.849965 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"thread_start_us":56,"threads_started":1}
I20260927 10:53:43.850112 22038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.845140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55632 (local address 127.14.68.60:35011)
0927 10:53:43.845351 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:43.845416 (+ 65us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:43.845444 (+ 28us) server_negotiation.cc:407] Connection header received
0927 10:53:43.845491 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:43.845493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:43.845535 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:43.845652 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:43.845898 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.846825 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.848432 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:43.849056 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:43.849167 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:43.849234 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:43.849922 (+ 688us) server_negotiation.cc:299] Negotiation successful
0927 10:53:43.849994 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":138,"thread_start_us":61,"threads_started":1}
I20260927 10:53:43.850057 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb6e814658944ac489f240027c3e09ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cf60ff31248349aea2c39fa73946403f" is_pre_election: true
I20260927 10:53:43.850297 21322 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb6e814658944ac489f240027c3e09ba in term 1.
I20260927 10:53:43.850607 21517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cf60ff31248349aea2c39fa73946403f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fb6e814658944ac489f240027c3e09ba" is_pre_election: true
I20260927 10:53:43.850665 21488 leader_election.cc:302] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf60ff31248349aea2c39fa73946403f, fb6e814658944ac489f240027c3e09ba; no voters: e37e387fd038459890cc0553983720ae
I20260927 10:53:43.850772 21517 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf60ff31248349aea2c39fa73946403f in term 1.
I20260927 10:53:43.850930 22030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 10:53:43.851009 22030 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 1 FOLLOWER]: Starting leader election (detected failure of leader e37e387fd038459890cc0553983720ae)
I20260927 10:53:43.851063 22030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:53:43.851250 21300 leader_election.cc:302] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf60ff31248349aea2c39fa73946403f, fb6e814658944ac489f240027c3e09ba; no voters: e37e387fd038459890cc0553983720ae
I20260927 10:53:43.851454 22031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 10:53:43.851509 22031 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 1 FOLLOWER]: Starting leader election (detected failure of leader e37e387fd038459890cc0553983720ae)
I20260927 10:53:43.851555 22031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:53:43.852312 22030 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: false } }
I20260927 10:53:43.852667 22031 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: false } }
I20260927 10:53:43.852784 22030 leader_election.cc:288] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [CANDIDATE]: Term 2 election: Requested vote from peers e37e387fd038459890cc0553983720ae (127.14.68.62:40605), cf60ff31248349aea2c39fa73946403f (127.14.68.61:33935)
I20260927 10:53:43.853087 22035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.852838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.0.0.1:47282)
0927 10:53:43.852946 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:43.852999 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 10:53:43.853078 22031 leader_election.cc:288] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [CANDIDATE]: Term 2 election: Requested vote from peers e37e387fd038459890cc0553983720ae (127.14.68.62:40605), fb6e814658944ac489f240027c3e09ba (127.14.68.60:35011)
I20260927 10:53:43.853322 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb6e814658944ac489f240027c3e09ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cf60ff31248349aea2c39fa73946403f"
I20260927 10:53:43.853499 21322 raft_consensus.cc:2379] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fb6e814658944ac489f240027c3e09ba in current term 2: Already voted for candidate cf60ff31248349aea2c39fa73946403f in this term.
I20260927 10:53:43.853811 22035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.853586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.0.0.1:47302)
0927 10:53:43.853697 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:43.853753 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 10:53:43.853940 22034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.853147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.0.0.1:47288)
0927 10:53:43.853809 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:43.853881 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":600}
I20260927 10:53:43.853940 21517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cf60ff31248349aea2c39fa73946403f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fb6e814658944ac489f240027c3e09ba"
I20260927 10:53:43.854102 21517 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cf60ff31248349aea2c39fa73946403f in current term 2: Already voted for candidate fb6e814658944ac489f240027c3e09ba in this term.
W20260927 10:53:43.854290 21490 leader_election.cc:334] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e37e387fd038459890cc0553983720ae (127.14.68.62:40605): Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
I20260927 10:53:43.854369 21490 leader_election.cc:302] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb6e814658944ac489f240027c3e09ba; no voters: cf60ff31248349aea2c39fa73946403f, e37e387fd038459890cc0553983720ae
I20260927 10:53:43.854499 22030 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 10:53:43.854672 22034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:43.854347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.0.0.1:47308)
0927 10:53:43.854558 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:43.854611 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260927 10:53:43.854786 21299 leader_election.cc:334] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e37e387fd038459890cc0553983720ae (127.14.68.62:40605): Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
I20260927 10:53:43.854892 21299 leader_election.cc:302] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf60ff31248349aea2c39fa73946403f; no voters: e37e387fd038459890cc0553983720ae, fb6e814658944ac489f240027c3e09ba
I20260927 10:53:43.855068 22031 raft_consensus.cc:2726] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 10:53:44.334247 22040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:44.333639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.2:47693)
0927 10:53:44.334018 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:44.334075 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":80,"threads_started":1}
I20260927 10:53:44.346284 22041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:44.345773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.1:38239)
0927 10:53:44.346104 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:44.346161 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":88,"threads_started":1}
I20260927 10:53:44.347982 22042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:44.347630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.3:50045)
0927 10:53:44.347860 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:44.347909 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":109,"threads_started":1}
I20260927 10:53:45.335387 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.334940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.2:41473)
0927 10:53:45.335230 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.335296 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":74,"threads_started":1}
I20260927 10:53:45.347242 22044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.346856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.1:54443)
0927 10:53:45.347091 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.347156 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":68,"threads_started":1}
I20260927 10:53:45.348867 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.348547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.3:41177)
0927 10:53:45.348743 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.348794 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":58,"threads_started":1}
I20260927 10:53:45.551249 22046 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:45.551390 22046 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: false } }
I20260927 10:53:45.551913 22046 leader_election.cc:288] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e37e387fd038459890cc0553983720ae (127.14.68.62:40605), cf60ff31248349aea2c39fa73946403f (127.14.68.61:33935)
I20260927 10:53:45.552485 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.552067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.0.0.1:47522)
0927 10:53:45.552310 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.552386 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":75,"threads_started":1}
I20260927 10:53:45.552464 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb6e814658944ac489f240027c3e09ba" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cf60ff31248349aea2c39fa73946403f" is_pre_election: true
I20260927 10:53:45.552623 21322 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb6e814658944ac489f240027c3e09ba in term 2.
I20260927 10:53:45.553004 21488 leader_election.cc:302] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf60ff31248349aea2c39fa73946403f, fb6e814658944ac489f240027c3e09ba; no voters:
I20260927 10:53:45.553225 22046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 10:53:45.553285 22046 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:45.553354 22046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 2 FOLLOWER]: Advancing to term 3
I20260927 10:53:45.553427 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.553151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.0.0.1:47538)
0927 10:53:45.553294 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.553356 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 10:53:45.553565 21490 leader_election.cc:334] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e37e387fd038459890cc0553983720ae (127.14.68.62:40605): Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
I20260927 10:53:45.554577 22046 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: false } }
I20260927 10:53:45.554924 22046 leader_election.cc:288] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [CANDIDATE]: Term 3 election: Requested vote from peers e37e387fd038459890cc0553983720ae (127.14.68.62:40605), cf60ff31248349aea2c39fa73946403f (127.14.68.61:33935)
I20260927 10:53:45.555187 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.554992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.0.0.1:47546)
0927 10:53:45.555080 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.555138 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 10:53:45.555348 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb6e814658944ac489f240027c3e09ba" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cf60ff31248349aea2c39fa73946403f"
I20260927 10:53:45.555500 21322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 2 FOLLOWER]: Advancing to term 3
I20260927 10:53:45.556020 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.555812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.0.0.1:47562)
0927 10:53:45.555916 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.555966 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 10:53:45.556138 21490 leader_election.cc:334] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e37e387fd038459890cc0553983720ae (127.14.68.62:40605): Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
I20260927 10:53:45.556783 21322 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb6e814658944ac489f240027c3e09ba in term 3.
I20260927 10:53:45.557047 21488 leader_election.cc:302] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf60ff31248349aea2c39fa73946403f, fb6e814658944ac489f240027c3e09ba; no voters: e37e387fd038459890cc0553983720ae
I20260927 10:53:45.557286 22046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 3 FOLLOWER]: Leader election won for term 3
I20260927 10:53:45.557478 22046 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [term 3 LEADER]: Becoming Leader. State: Replica: fb6e814658944ac489f240027c3e09ba, State: Running, Role: LEADER
I20260927 10:53:45.557770 22046 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: false } }
I20260927 10:53:45.558799 22049 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb6e814658944ac489f240027c3e09ba. Latest consensus state: current_term: 3 leader_uuid: "fb6e814658944ac489f240027c3e09ba" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: false } } }
I20260927 10:53:45.558917 22049 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:45.559214 22051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:45.561517 22051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:45.561905 22051 catalog_manager.cc:1259] Loaded cluster ID: a738ba3f15fa434495982601af43bfd6
I20260927 10:53:45.561951 22051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:45.562502 22051 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:45.562884 22051 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba: Loaded TSK: 0
I20260927 10:53:45.563566 22051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:53:45.595464 21507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:53:45.598754 21507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:45.601778 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.601480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.0.0.1:47576)
0927 10:53:45.601609 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.601698 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 10:53:45.601850 21322 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [term 3 FOLLOWER]: Refusing update from remote peer fb6e814658944ac489f240027c3e09ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 10:53:45.602404 22049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 10:53:45.603056 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.602847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.0.0.1:47578)
0927 10:53:45.602951 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.603000 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 10:53:45.604012 21490 consensus_peers.cc:487] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba -> Peer e37e387fd038459890cc0553983720ae (127.14.68.62:40605): Couldn't send request to peer e37e387fd038459890cc0553983720ae. Status: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 10:53:45.604100 22053 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb6e814658944ac489f240027c3e09ba. Latest consensus state: current_term: 3 leader_uuid: "fb6e814658944ac489f240027c3e09ba" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: false } } }
I20260927 10:53:45.604264 22053 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:53:45.605122 22052 mvcc.cc:205] Tried to move back new op lower bound from 7333914319260418048 to 7333914319086817280. Current Snapshot: MvccSnapshot[applied={T|T < 7333914319260418048}]
I20260927 10:53:45.605377 22046 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "fb6e814658944ac489f240027c3e09ba" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: false } } }
I20260927 10:53:45.605499 22046 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:45.605854 22053 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "fb6e814658944ac489f240027c3e09ba" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e37e387fd038459890cc0553983720ae" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40605 } } peers { permanent_uuid: "cf60ff31248349aea2c39fa73946403f" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "fb6e814658944ac489f240027c3e09ba" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 35011 } attrs { promote: false } } }
I20260927 10:53:45.606000 22053 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf60ff31248349aea2c39fa73946403f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:53:45.606338 22054 mvcc.cc:205] Tried to move back new op lower bound from 7333914319260418048 to 7333914319086817280. Current Snapshot: MvccSnapshot[applied={T|T < 7333914319260418048}]
I20260927 10:53:45.615269 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.611073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:36575 (local address 127.0.0.1:46952)
0927 10:53:45.611249 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.611264 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:45.611355 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:45.611880 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:45.611883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:45.611898 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:45.611994 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:45.612001 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:45.612974 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:45.612980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:45.614151 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:45.614163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:45.614856 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:45.614860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:45.614947 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:45.614970 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:53:45.615076 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 10:53:45.615144 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 10:53:45.615293 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.611191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46952 (local address 127.14.68.3:36575)
0927 10:53:45.611572 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:45.611579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:45.611595 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:53:45.611653 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:45.611656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:45.611752 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:45.611872 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:45.612129 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:45.612871 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:45.614271 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:45.614744 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:45.614856 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:45.614922 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:45.615094 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 10:53:45.615173 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":82,"threads_started":1}
I20260927 10:53:45.615939 21889 tablet_service.cc:1467] Processing CreateTablet for tablet e6e611b0ce654429b7ee2d144b8ab543 (DEFAULT_TABLE table=test-table [id=9505fc3062b24a148ba2be5c41978621]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:45.616261 21889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e611b0ce654429b7ee2d144b8ab543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:45.616725 22061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.611661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:43831 (local address 127.0.0.1:56256)
0927 10:53:45.612350 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.612365 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:45.612485 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:45.613009 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:45.613012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:45.613028 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:45.613125 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:45.613130 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:45.614118 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:45.614122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:45.615349 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:45.615359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:45.616309 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:45.616312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:45.616400 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:45.616421 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:53:45.616503 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 10:53:45.616575 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":109,"threads_started":1}
I20260927 10:53:45.617070 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.612186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56256 (local address 127.14.68.2:43831)
0927 10:53:45.612716 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:45.612723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:45.612736 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:53:45.612781 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:45.612784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:45.612827 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:45.612908 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:45.613244 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:45.613995 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:45.615460 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:45.616061 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:45.616181 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:45.616245 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:45.616867 (+ 622us) server_negotiation.cc:299] Negotiation successful
0927 10:53:45.616943 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":87,"threads_started":1}
I20260927 10:53:45.617566 21754 tablet_service.cc:1467] Processing CreateTablet for tablet e6e611b0ce654429b7ee2d144b8ab543 (DEFAULT_TABLE table=test-table [id=9505fc3062b24a148ba2be5c41978621]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:45.617858 21754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e611b0ce654429b7ee2d144b8ab543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:45.618818 22064 tablet_bootstrap.cc:492] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e: Bootstrap starting.
I20260927 10:53:45.619772 22064 tablet_bootstrap.cc:654] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:45.619801 22062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.611904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.14.68.1:36925)
0927 10:53:45.612140 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:45.612145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:45.612160 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:53:45.612293 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:45.612297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:45.612344 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:45.612420 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:45.612762 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:45.613435 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:45.615509 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:45.619185 (+ 3676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:45.619336 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:45.619398 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:45.619572 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 10:53:45.619649 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":57,"threads_started":1}
I20260927 10:53:45.619786 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.611518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:36925 (local address 127.0.0.1:50450)
0927 10:53:45.612080 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.612096 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:45.612170 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:45.612516 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:45.612519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:45.612534 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:45.612629 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:45.612634 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:45.613545 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:45.613548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:45.614513 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:45.614523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:45.619335 (+ 4812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:45.619338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:45.619453 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:45.619469 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 10:53:45.619556 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 10:53:45.619623 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":100,"threads_started":1}
I20260927 10:53:45.620532 21617 tablet_service.cc:1467] Processing CreateTablet for tablet e6e611b0ce654429b7ee2d144b8ab543 (DEFAULT_TABLE table=test-table [id=9505fc3062b24a148ba2be5c41978621]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:45.620816 21617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e611b0ce654429b7ee2d144b8ab543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:45.620976 22066 tablet_bootstrap.cc:492] T e6e611b0ce654429b7ee2d144b8ab543 P 0a59c8b46a2b48c185bfc0cd886caa51: Bootstrap starting.
I20260927 10:53:45.621486 22064 tablet_bootstrap.cc:492] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e: No bootstrap required, opened a new log
I20260927 10:53:45.621582 22064 ts_tablet_manager.cc:1397] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 10:53:45.621830 22066 tablet_bootstrap.cc:654] T e6e611b0ce654429b7ee2d144b8ab543 P 0a59c8b46a2b48c185bfc0cd886caa51: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:45.621951 22064 raft_consensus.cc:348] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198bfa5be48e491eb86d59eab104046e" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36575 } } peers { permanent_uuid: "8f96cff0ec274e9eac28349a5a876141" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36925 } } peers { permanent_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43831 } }
I20260927 10:53:45.622056 22064 raft_consensus.cc:374] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:45.622107 22064 raft_consensus.cc:729] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 198bfa5be48e491eb86d59eab104046e, State: Initialized, Role: FOLLOWER
I20260927 10:53:45.622258 22064 consensus_queue.cc:260] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198bfa5be48e491eb86d59eab104046e" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36575 } } peers { permanent_uuid: "8f96cff0ec274e9eac28349a5a876141" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36925 } } peers { permanent_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43831 } }
I20260927 10:53:45.622661 22064 ts_tablet_manager.cc:1428] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 10:53:45.623476 22066 tablet_bootstrap.cc:492] T e6e611b0ce654429b7ee2d144b8ab543 P 0a59c8b46a2b48c185bfc0cd886caa51: No bootstrap required, opened a new log
I20260927 10:53:45.623608 22066 ts_tablet_manager.cc:1397] T e6e611b0ce654429b7ee2d144b8ab543 P 0a59c8b46a2b48c185bfc0cd886caa51: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:53:45.623850 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.623527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.3:38393)
0927 10:53:45.623725 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.623783 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 10:53:45.624197 22066 raft_consensus.cc:348] T e6e611b0ce654429b7ee2d144b8ab543 P 0a59c8b46a2b48c185bfc0cd886caa51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198bfa5be48e491eb86d59eab104046e" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36575 } } peers { permanent_uuid: "8f96cff0ec274e9eac28349a5a876141" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36925 } } peers { permanent_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43831 } }
I20260927 10:53:45.624313 22066 raft_consensus.cc:374] T e6e611b0ce654429b7ee2d144b8ab543 P 0a59c8b46a2b48c185bfc0cd886caa51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:45.624351 22066 raft_consensus.cc:729] T e6e611b0ce654429b7ee2d144b8ab543 P 0a59c8b46a2b48c185bfc0cd886caa51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a59c8b46a2b48c185bfc0cd886caa51, State: Initialized, Role: FOLLOWER
I20260927 10:53:45.624487 22066 consensus_queue.cc:260] T e6e611b0ce654429b7ee2d144b8ab543 P 0a59c8b46a2b48c185bfc0cd886caa51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198bfa5be48e491eb86d59eab104046e" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36575 } } peers { permanent_uuid: "8f96cff0ec274e9eac28349a5a876141" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36925 } } peers { permanent_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43831 } }
I20260927 10:53:45.624645 22069 tablet_bootstrap.cc:492] T e6e611b0ce654429b7ee2d144b8ab543 P 8f96cff0ec274e9eac28349a5a876141: Bootstrap starting.
I20260927 10:53:45.624833 22066 ts_tablet_manager.cc:1428] T e6e611b0ce654429b7ee2d144b8ab543 P 0a59c8b46a2b48c185bfc0cd886caa51: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:53:45.625510 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.625225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.2:41287)
0927 10:53:45.625396 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.625445 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260927 10:53:45.625607 22069 tablet_bootstrap.cc:654] T e6e611b0ce654429b7ee2d144b8ab543 P 8f96cff0ec274e9eac28349a5a876141: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:45.626569 21956 heartbeater.cc:503] Master 127.14.68.60:35011 was elected leader, sending a full tablet report...
I20260927 10:53:45.626745 21820 heartbeater.cc:503] Master 127.14.68.60:35011 was elected leader, sending a full tablet report...
I20260927 10:53:45.627617 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.627337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.3:56713)
0927 10:53:45.627490 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.627556 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 10:53:45.628099 22069 tablet_bootstrap.cc:492] T e6e611b0ce654429b7ee2d144b8ab543 P 8f96cff0ec274e9eac28349a5a876141: No bootstrap required, opened a new log
I20260927 10:53:45.628181 22069 ts_tablet_manager.cc:1397] T e6e611b0ce654429b7ee2d144b8ab543 P 8f96cff0ec274e9eac28349a5a876141: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 10:53:45.628532 22069 raft_consensus.cc:348] T e6e611b0ce654429b7ee2d144b8ab543 P 8f96cff0ec274e9eac28349a5a876141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198bfa5be48e491eb86d59eab104046e" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36575 } } peers { permanent_uuid: "8f96cff0ec274e9eac28349a5a876141" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36925 } } peers { permanent_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43831 } }
I20260927 10:53:45.628615 22069 raft_consensus.cc:374] T e6e611b0ce654429b7ee2d144b8ab543 P 8f96cff0ec274e9eac28349a5a876141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:45.628677 22069 raft_consensus.cc:729] T e6e611b0ce654429b7ee2d144b8ab543 P 8f96cff0ec274e9eac28349a5a876141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f96cff0ec274e9eac28349a5a876141, State: Initialized, Role: FOLLOWER
I20260927 10:53:45.628793 22069 consensus_queue.cc:260] T e6e611b0ce654429b7ee2d144b8ab543 P 8f96cff0ec274e9eac28349a5a876141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198bfa5be48e491eb86d59eab104046e" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36575 } } peers { permanent_uuid: "8f96cff0ec274e9eac28349a5a876141" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36925 } } peers { permanent_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43831 } }
I20260927 10:53:45.629093 22069 ts_tablet_manager.cc:1428] T e6e611b0ce654429b7ee2d144b8ab543 P 8f96cff0ec274e9eac28349a5a876141: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:53:45.629644 22044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.629400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.1:60233)
0927 10:53:45.629530 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.629578 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 10:53:45.629760 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.629504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.2:40393)
0927 10:53:45.629641 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.629697 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 10:53:45.630540 21683 heartbeater.cc:503] Master 127.14.68.60:35011 was elected leader, sending a full tablet report...
I20260927 10:53:45.630693 22044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.630127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.1:34441)
0927 10:53:45.630497 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.630547 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297}
I20260927 10:53:45.654155 22068 raft_consensus.cc:482] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:45.654291 22068 raft_consensus.cc:504] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198bfa5be48e491eb86d59eab104046e" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36575 } } peers { permanent_uuid: "8f96cff0ec274e9eac28349a5a876141" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36925 } } peers { permanent_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43831 } }
I20260927 10:53:45.654736 22068 leader_election.cc:288] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f96cff0ec274e9eac28349a5a876141 (127.14.68.1:36925), 0a59c8b46a2b48c185bfc0cd886caa51 (127.14.68.2:43831)
I20260927 10:53:45.655169 21774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e611b0ce654429b7ee2d144b8ab543" candidate_uuid: "198bfa5be48e491eb86d59eab104046e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" is_pre_election: true
I20260927 10:53:45.655225 21637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e611b0ce654429b7ee2d144b8ab543" candidate_uuid: "198bfa5be48e491eb86d59eab104046e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f96cff0ec274e9eac28349a5a876141" is_pre_election: true
I20260927 10:53:45.655335 21774 raft_consensus.cc:2454] T e6e611b0ce654429b7ee2d144b8ab543 P 0a59c8b46a2b48c185bfc0cd886caa51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 198bfa5be48e491eb86d59eab104046e in term 0.
I20260927 10:53:45.655395 21637 raft_consensus.cc:2454] T e6e611b0ce654429b7ee2d144b8ab543 P 8f96cff0ec274e9eac28349a5a876141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 198bfa5be48e491eb86d59eab104046e in term 0.
I20260927 10:53:45.655752 21843 leader_election.cc:302] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a59c8b46a2b48c185bfc0cd886caa51, 198bfa5be48e491eb86d59eab104046e; no voters:
I20260927 10:53:45.655969 22068 raft_consensus.cc:2781] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:45.656029 22068 raft_consensus.cc:482] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:45.656057 22068 raft_consensus.cc:3037] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:45.657143 22068 raft_consensus.cc:504] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198bfa5be48e491eb86d59eab104046e" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36575 } } peers { permanent_uuid: "8f96cff0ec274e9eac28349a5a876141" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36925 } } peers { permanent_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43831 } }
I20260927 10:53:45.657503 22068 leader_election.cc:288] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e [CANDIDATE]: Term 1 election: Requested vote from peers 8f96cff0ec274e9eac28349a5a876141 (127.14.68.1:36925), 0a59c8b46a2b48c185bfc0cd886caa51 (127.14.68.2:43831)
I20260927 10:53:45.657857 21637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e611b0ce654429b7ee2d144b8ab543" candidate_uuid: "198bfa5be48e491eb86d59eab104046e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f96cff0ec274e9eac28349a5a876141"
I20260927 10:53:45.657863 21774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e611b0ce654429b7ee2d144b8ab543" candidate_uuid: "198bfa5be48e491eb86d59eab104046e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a59c8b46a2b48c185bfc0cd886caa51"
I20260927 10:53:45.657956 21637 raft_consensus.cc:3037] T e6e611b0ce654429b7ee2d144b8ab543 P 8f96cff0ec274e9eac28349a5a876141 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:45.657986 21774 raft_consensus.cc:3037] T e6e611b0ce654429b7ee2d144b8ab543 P 0a59c8b46a2b48c185bfc0cd886caa51 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:45.658835 21637 raft_consensus.cc:2454] T e6e611b0ce654429b7ee2d144b8ab543 P 8f96cff0ec274e9eac28349a5a876141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 198bfa5be48e491eb86d59eab104046e in term 1.
I20260927 10:53:45.658936 21774 raft_consensus.cc:2454] T e6e611b0ce654429b7ee2d144b8ab543 P 0a59c8b46a2b48c185bfc0cd886caa51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 198bfa5be48e491eb86d59eab104046e in term 1.
I20260927 10:53:45.659106 21845 leader_election.cc:302] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 198bfa5be48e491eb86d59eab104046e, 8f96cff0ec274e9eac28349a5a876141; no voters:
I20260927 10:53:45.659274 22068 raft_consensus.cc:2781] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:45.659510 22068 raft_consensus.cc:686] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e [term 1 LEADER]: Becoming Leader. State: Replica: 198bfa5be48e491eb86d59eab104046e, State: Running, Role: LEADER
I20260927 10:53:45.659678 22068 consensus_queue.cc:237] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198bfa5be48e491eb86d59eab104046e" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36575 } } peers { permanent_uuid: "8f96cff0ec274e9eac28349a5a876141" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36925 } } peers { permanent_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43831 } }
I20260927 10:53:45.660240 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.659981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.3:55045)
0927 10:53:45.660083 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.660143 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 10:53:45.661360 21506 catalog_manager.cc:5495] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e reported cstate change: term changed from 0 to 1, leader changed from <none> to 198bfa5be48e491eb86d59eab104046e (127.14.68.3). New cstate: current_term: 1 leader_uuid: "198bfa5be48e491eb86d59eab104046e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198bfa5be48e491eb86d59eab104046e" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 36575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f96cff0ec274e9eac28349a5a876141" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43831 } health_report { overall_health: UNKNOWN } } }
I20260927 10:53:45.682377 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.679417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46966 (local address 127.14.68.3:36575)
0927 10:53:45.679524 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:45.679528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:45.679541 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:53:45.679674 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:45.679676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:45.679778 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:45.679839 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:45.680112 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:45.680633 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:45.681490 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:45.681708 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:45.681787 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:45.682020 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:45.682085 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:45.682136 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:45.682228 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 10:53:45.682271 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 10:53:45.684444 21774 raft_consensus.cc:1261] T e6e611b0ce654429b7ee2d144b8ab543 P 0a59c8b46a2b48c185bfc0cd886caa51 [term 1 FOLLOWER]: Refusing update from remote peer 198bfa5be48e491eb86d59eab104046e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:45.684444 21637 raft_consensus.cc:1261] T e6e611b0ce654429b7ee2d144b8ab543 P 8f96cff0ec274e9eac28349a5a876141 [term 1 FOLLOWER]: Refusing update from remote peer 198bfa5be48e491eb86d59eab104046e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:45.684950 22073 consensus_queue.cc:1035] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f96cff0ec274e9eac28349a5a876141" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:45.685127 22068 consensus_queue.cc:1035] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a59c8b46a2b48c185bfc0cd886caa51" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:45.687167 22044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.686787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.1:34063)
0927 10:53:45.687030 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.687093 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260927 10:53:45.687484 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.687028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.2:55065)
0927 10:53:45.687342 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.687405 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245}
I20260927 10:53:45.691759 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.691483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.3:34183)
0927 10:53:45.691596 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.691655 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 10:53:45.693393 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.692937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.2:38549)
0927 10:53:45.693259 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.693315 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249}
I20260927 10:53:45.694193 22044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.693934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40605 (local address 127.14.68.1:48393)
0927 10:53:45.694090 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.694137 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 10:53:45.696410 21506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:45.696566 21506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:45.700273 21506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb6e814658944ac489f240027c3e09ba: Sending DeleteTablet for 3 replicas of tablet e6e611b0ce654429b7ee2d144b8ab543
I20260927 10:53:45.700887 21889 tablet_service.cc:1514] Processing DeleteTablet for tablet e6e611b0ce654429b7ee2d144b8ab543 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:45 UTC) from {username='oryx', principal='oryx/127.14.68.60@KRBTEST.COM'} at 127.0.0.1:46952
I20260927 10:53:45.701031 21617 tablet_service.cc:1514] Processing DeleteTablet for tablet e6e611b0ce654429b7ee2d144b8ab543 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:45 UTC) from {username='oryx', principal='oryx/127.14.68.60@KRBTEST.COM'} at 127.0.0.1:50450
I20260927 10:53:45.701071 21754 tablet_service.cc:1514] Processing DeleteTablet for tablet e6e611b0ce654429b7ee2d144b8ab543 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:45 UTC) from {username='oryx', principal='oryx/127.14.68.60@KRBTEST.COM'} at 127.0.0.1:56256
I20260927 10:53:45.701269 22084 tablet_replica.cc:330] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e: stopping tablet replica
I20260927 10:53:45.701393 22086 tablet_replica.cc:330] T e6e611b0ce654429b7ee2d144b8ab543 P 0a59c8b46a2b48c185bfc0cd886caa51: stopping tablet replica
I20260927 10:53:45.701407 22084 raft_consensus.cc:2229] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:53:45.701591 22085 tablet_replica.cc:330] T e6e611b0ce654429b7ee2d144b8ab543 P 8f96cff0ec274e9eac28349a5a876141: stopping tablet replica
I20260927 10:53:45.701602 22084 raft_consensus.cc:2258] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:45.701848 22085 raft_consensus.cc:2229] T e6e611b0ce654429b7ee2d144b8ab543 P 8f96cff0ec274e9eac28349a5a876141 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:45.701903 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 21457
I20260927 10:53:45.703899 22086 raft_consensus.cc:2229] T e6e611b0ce654429b7ee2d144b8ab543 P 0a59c8b46a2b48c185bfc0cd886caa51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:45.704061 22086 raft_consensus.cc:2258] T e6e611b0ce654429b7ee2d144b8ab543 P 0a59c8b46a2b48c185bfc0cd886caa51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:45.704218 22084 ts_tablet_manager.cc:1905] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:45.704854 22086 ts_tablet_manager.cc:1905] T e6e611b0ce654429b7ee2d144b8ab543 P 0a59c8b46a2b48c185bfc0cd886caa51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:45.706733 22084 ts_tablet_manager.cc:1918] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:45.706830 22084 log.cc:1196] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/e6e611b0ce654429b7ee2d144b8ab543
I20260927 10:53:45.707109 22084 ts_tablet_manager.cc:1939] T e6e611b0ce654429b7ee2d144b8ab543 P 198bfa5be48e491eb86d59eab104046e: Deleting consensus metadata
I20260927 10:53:45.707834 21490 catalog_manager.cc:4841] TS 198bfa5be48e491eb86d59eab104046e (127.14.68.3:36575): tablet e6e611b0ce654429b7ee2d144b8ab543 (table test-table [id=9505fc3062b24a148ba2be5c41978621]) successfully deleted
I20260927 10:53:45.708173 22086 ts_tablet_manager.cc:1918] T e6e611b0ce654429b7ee2d144b8ab543 P 0a59c8b46a2b48c185bfc0cd886caa51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:45.708259 22086 log.cc:1196] T e6e611b0ce654429b7ee2d144b8ab543 P 0a59c8b46a2b48c185bfc0cd886caa51: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/e6e611b0ce654429b7ee2d144b8ab543
I20260927 10:53:45.708549 22086 ts_tablet_manager.cc:1939] T e6e611b0ce654429b7ee2d144b8ab543 P 0a59c8b46a2b48c185bfc0cd886caa51: Deleting consensus metadata
I20260927 10:53:45.709259 21488 catalog_manager.cc:4841] TS 0a59c8b46a2b48c185bfc0cd886caa51 (127.14.68.2:43831): tablet e6e611b0ce654429b7ee2d144b8ab543 (table test-table [id=9505fc3062b24a148ba2be5c41978621]) successfully deleted
I20260927 10:53:45.710155 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 21691
I20260927 10:53:45.710966 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:45.710254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:36925 (local address 127.0.0.1:50452)
0927 10:53:45.710802 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:45.710863 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.1:36925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452}
W20260927 10:53:45.711295 21490 catalog_manager.cc:4584] TS 8f96cff0ec274e9eac28349a5a876141 (127.14.68.1:36925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e6e611b0ce654429b7ee2d144b8ab543: Network error: Client connection negotiation failed: client connection to 127.14.68.1:36925: connect: Connection refused (error 111)
I20260927 10:53:45.716472 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 21827
I20260927 10:53:45.724802 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 21177
I20260927 10:53:45.730441 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 21285
2026-09-27T10:53:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7248 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:53:45 dist-test-slave-tz51 krb5kdc[22089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:53:45 dist-test-slave-tz51 krb5kdc[22089](info): set up 2 sockets
Sep 27 10:53:45 dist-test-slave-tz51 krb5kdc[22089](info): commencing operation
krb5kdc: starting...
W20260927 10:53:46.776373 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:53:46 dist-test-slave-tz51 krb5kdc[22089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:53:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.62.keytab.
Entry for principal oryx/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.62.keytab.
I20260927 10:53:46.900887 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:40641
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.62.keytab
--principal=oryx/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:40641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:47.010128 22102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:47.010421 22102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:47.010491 22102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:47.014528 22102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:47.014617 22102 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 10:53:47.014652 22102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:47.014694 22102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:47.014725 22102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:47.014760 22102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:47.020087 22102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:40641
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.62.keytab
--principal=oryx/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:40641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:47.021186 22102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:47.022254 22102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:47.030670 22107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:47.030719 22108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:47.030670 22110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:47.030898 22102 server_base.cc:1034] running on GCE node
I20260927 10:53:47.031467 22102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:47.032670 22102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:47.033841 22102 hybrid_clock.cc:648] HybridClock initialized: now 1790506427033810 us; error 44 us; skew 500 ppm
Sep 27 10:53:47 dist-test-slave-tz51 krb5kdc[22089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506427, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:47.036655 22102 init.cc:377] Logged in from keytab as oryx/127.14.68.62@KRBTEST.COM (short username oryx)
I20260927 10:53:47.037497 22102 webserver.cc:460] Webserver started at http://127.14.68.62:37505/ using document root <none> and password file <none>
I20260927 10:53:47.038015 22102 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:47.038071 22102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:47.038233 22102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:47.040285 22102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "abc739bf6a844b7db4a2d03c71ef39cc"
format_stamp: "Formatted at 2026-09-27 10:53:47 on dist-test-slave-tz51"
server_key: "95d6b7eecdabd33d143373442cd0d6c8"
server_key_iv: "25ef138429d8afeadfc11e9f7df1119b"
server_key_version: "encryptionkey@0"
I20260927 10:53:47.040812 22102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "abc739bf6a844b7db4a2d03c71ef39cc"
format_stamp: "Formatted at 2026-09-27 10:53:47 on dist-test-slave-tz51"
server_key: "95d6b7eecdabd33d143373442cd0d6c8"
server_key_iv: "25ef138429d8afeadfc11e9f7df1119b"
server_key_version: "encryptionkey@0"
I20260927 10:53:47.044310 22102 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:53:47.046655 22117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:47.047973 22102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:47.048101 22102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "abc739bf6a844b7db4a2d03c71ef39cc"
format_stamp: "Formatted at 2026-09-27 10:53:47 on dist-test-slave-tz51"
server_key: "95d6b7eecdabd33d143373442cd0d6c8"
server_key_iv: "25ef138429d8afeadfc11e9f7df1119b"
server_key_version: "encryptionkey@0"
I20260927 10:53:47.048216 22102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:47.060425 22102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:47.063082 22102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:47.063280 22102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:47.071007 22102 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:40641
I20260927 10:53:47.071033 22179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:40641 every 8 connection(s)
I20260927 10:53:47.071919 22102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:47.074491 22180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:47.076973 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 22102
I20260927 10:53:47.077128 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:53:47.077433 14608 external_mini_cluster.cc:1428] Setting key bffc9dc4e781f9173e19596e06fafce2
I20260927 10:53:47.080510 22180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc: Bootstrap starting.
I20260927 10:53:47.082630 22180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:47.083340 22180 log.cc:824] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:47.085217 22180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc: No bootstrap required, opened a new log
Sep 27 10:53:47 dist-test-slave-tz51 krb5kdc[22089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM
I20260927 10:53:47.088830 22180 raft_consensus.cc:348] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc739bf6a844b7db4a2d03c71ef39cc" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40641 } }
I20260927 10:53:47.089078 22180 raft_consensus.cc:374] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:47.089146 22180 raft_consensus.cc:729] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abc739bf6a844b7db4a2d03c71ef39cc, State: Initialized, Role: FOLLOWER
I20260927 10:53:47.089617 22180 consensus_queue.cc:260] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc739bf6a844b7db4a2d03c71ef39cc" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40641 } }
I20260927 10:53:47.089766 22180 raft_consensus.cc:388] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:47.089838 22180 raft_consensus.cc:482] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:47.089957 22180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:47.090885 22180 raft_consensus.cc:504] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc739bf6a844b7db4a2d03c71ef39cc" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40641 } }
I20260927 10:53:47.091058 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.078832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53698 (local address 127.14.68.62:40641)
0927 10:53:47.079243 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.079257 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.079291 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:53:47.079964 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.079982 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.080307 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.080633 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:47.081453 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.082429 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.083342 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.083915 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.084081 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.086511 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:47.086516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:47.086529 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:47.086555 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:47.088282 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:47.088731 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:47.088736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:47.088740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:47.088845 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:47.089124 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:47.089130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:47.089133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:47.089589 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:47.089798 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.090061 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.090340 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":111,"threads_started":1}
I20260927 10:53:47.091274 22180 leader_election.cc:302] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abc739bf6a844b7db4a2d03c71ef39cc; no voters:
I20260927 10:53:47.091589 22180 leader_election.cc:288] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:47.091727 22185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:47.091933 22185 raft_consensus.cc:686] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc [term 1 LEADER]: Becoming Leader. State: Replica: abc739bf6a844b7db4a2d03c71ef39cc, State: Running, Role: LEADER
I20260927 10:53:47.092221 22185 consensus_queue.cc:237] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc739bf6a844b7db4a2d03c71ef39cc" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40641 } }
I20260927 10:53:47.092839 22180 sys_catalog.cc:564] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:47.093269 22185 sys_catalog.cc:455] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader abc739bf6a844b7db4a2d03c71ef39cc. Latest consensus state: current_term: 1 leader_uuid: "abc739bf6a844b7db4a2d03c71ef39cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc739bf6a844b7db4a2d03c71ef39cc" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40641 } } }
I20260927 10:53:47.093380 22185 sys_catalog.cc:458] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:47.093746 22189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:47.094138 22186 sys_catalog.cc:455] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abc739bf6a844b7db4a2d03c71ef39cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc739bf6a844b7db4a2d03c71ef39cc" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40641 } } }
I20260927 10:53:47.094242 22186 sys_catalog.cc:458] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:47.096801 22189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:47.102331 22189 catalog_manager.cc:1347] Generated new cluster ID: af617172e9e542d0b88242be42ddecb0
I20260927 10:53:47.102399 22189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:47.109580 22189 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:47.110728 22189 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:47.118511 22189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc: Generated new TSK 0
I20260927 10:53:47.119220 22189 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.1.keytab.
Entry for principal oryx/127.14.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.1.keytab.
I20260927 10:53:47.152760 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:40641
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.1.keytab
--principal=oryx/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:47.281401 22206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:47.281637 22206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:47.281680 22206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:47.285552 22206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:47.285600 22206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:47.285637 22206 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:53:47.285667 22206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:53:47.290596 22206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.1.keytab
--principal=oryx/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.68.62:40641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:47.291494 22206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:47.292554 22206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:47.301967 22211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:47.302059 22206 server_base.cc:1034] running on GCE node
W20260927 10:53:47.301963 22212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:47.301961 22214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:47.302697 22206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:47.303287 22206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:47.304440 22206 hybrid_clock.cc:648] HybridClock initialized: now 1790506427304427 us; error 23 us; skew 500 ppm
Sep 27 10:53:47 dist-test-slave-tz51 krb5kdc[22089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506427, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:47.307366 22206 init.cc:377] Logged in from keytab as oryx/127.14.68.1@KRBTEST.COM (short username oryx)
I20260927 10:53:47.308271 22206 webserver.cc:460] Webserver started at http://127.14.68.1:33347/ using document root <none> and password file <none>
I20260927 10:53:47.308887 22206 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:47.308974 22206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:47.309258 22206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:47.311285 22206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "5729ed1c02e7442fb08b248c96e9f55a"
format_stamp: "Formatted at 2026-09-27 10:53:47 on dist-test-slave-tz51"
server_key: "06e62f0ec6f5f3983492c904865b0790"
server_key_iv: "0dbf92c7e8ff5d927b64c9580a980104"
server_key_version: "encryptionkey@0"
I20260927 10:53:47.311851 22206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "5729ed1c02e7442fb08b248c96e9f55a"
format_stamp: "Formatted at 2026-09-27 10:53:47 on dist-test-slave-tz51"
server_key: "06e62f0ec6f5f3983492c904865b0790"
server_key_iv: "0dbf92c7e8ff5d927b64c9580a980104"
server_key_version: "encryptionkey@0"
I20260927 10:53:47.315502 22206 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:53:47.317830 22221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:47.319094 22206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:47.319195 22206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "5729ed1c02e7442fb08b248c96e9f55a"
format_stamp: "Formatted at 2026-09-27 10:53:47 on dist-test-slave-tz51"
server_key: "06e62f0ec6f5f3983492c904865b0790"
server_key_iv: "0dbf92c7e8ff5d927b64c9580a980104"
server_key_version: "encryptionkey@0"
I20260927 10:53:47.319278 22206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:47.333153 22206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:47.335795 22206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:47.335932 22206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:47.337173 22206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:47.337218 22206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:47.337280 22206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:47.337311 22206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:53:47 dist-test-slave-tz51 krb5kdc[22089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506427, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM
I20260927 10:53:47.348137 22206 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:35727
I20260927 10:53:47.348201 22336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:35727 every 8 connection(s)
I20260927 10:53:47.349130 22206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:53:47.349375 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 22206
I20260927 10:53:47.349534 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:53:47.349890 14608 external_mini_cluster.cc:1428] Setting key 2ccc0524ecdfd9b21eb8e32eac712dba
I20260927 10:53:47.352537 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.338053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:51175 (local address 127.14.68.62:40641)
0927 10:53:47.338174 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.338178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.339264 (+ 1086us) server_negotiation.cc:407] Connection header received
0927 10:53:47.340408 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.340412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.340472 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.340564 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:47.342159 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.343064 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.345011 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.345333 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.345422 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.348502 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:47.348506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:47.348509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:47.348536 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:47.350225 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:47.350832 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:47.350835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:47.350837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:47.350920 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:47.351320 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:47.351324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:47.351326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:47.351551 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:47.351629 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.352255 (+ 626us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.352401 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:53:47.353572 22230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.338561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40641 (local address 127.14.68.1:51175)
0927 10:53:47.339118 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:47.339166 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:47.340166 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:47.340778 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:47.340789 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:47.341281 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:47.341984 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.342007 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.343219 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.343227 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.344868 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.344883 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.345506 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.345511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.345741 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.346420 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:47.346444 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:47.348331 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:47.350345 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:47.350350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:47.350363 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:47.350711 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:47.351019 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:47.351021 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:47.351022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:47.351205 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:47.351662 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:47.351668 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:47.351983 (+ 315us) client_negotiation.cc:769] Sending connection context
0927 10:53:47.352190 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 10:53:47.352541 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":137,"threads_started":1}
I20260927 10:53:47.354838 22337 heartbeater.cc:347] Connected to a master server at 127.14.68.62:40641
I20260927 10:53:47.354979 22337 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:47.355523 22337 heartbeater.cc:511] Master 127.14.68.62:40641 requested a full tablet report, sending...
I20260927 10:53:47.357228 22134 ts_manager.cc:194] Registered new tserver with Master: 5729ed1c02e7442fb08b248c96e9f55a (127.14.68.1:35727)
I20260927 10:53:47.358770 22134 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:51175
WARNING: no policy specified for oryx/127.14.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.68.2@KRBTEST.COM" created.
I20260927 10:53:47.363504 22344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.356782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40641 (local address 127.14.68.1:43007)
0927 10:53:47.357068 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:47.357078 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:47.357183 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:47.357456 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:47.357459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:47.357604 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:47.357687 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.357691 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.358395 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.358398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.359012 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.359020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.359540 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.359547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.359658 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.360214 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:47.360229 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:47.360599 (+ 370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:47.362193 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:47.362195 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:47.362196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:47.362372 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:47.362606 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:47.362608 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:47.362610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:47.362679 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:47.363126 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:47.363130 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:47.363204 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 10:53:47.363288 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 10:53:47.363382 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":93,"threads_started":1}
I20260927 10:53:47.363734 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.356832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:43007 (local address 127.14.68.62:40641)
0927 10:53:47.356940 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.356944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.357149 (+ 205us) server_negotiation.cc:407] Connection header received
0927 10:53:47.357254 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.357257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.357295 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.357359 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:47.357796 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.358276 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.359122 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.359391 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.359505 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.360752 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:47.360755 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:47.360758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:47.360779 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:47.362019 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:47.362456 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:47.362458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:47.362459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:47.362528 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:47.362776 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:47.362780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:47.362782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:47.363011 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:47.363128 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.363374 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.363496 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.2.keytab.
Entry for principal oryx/127.14.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.2.keytab.
I20260927 10:53:47.377920 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:40641
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.2.keytab
--principal=oryx/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:47.482959 22346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:47.483233 22346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:47.483309 22346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:47.487254 22346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:47.487345 22346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:47.487417 22346 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:53:47.487495 22346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:53:47.492486 22346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.2.keytab
--principal=oryx/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.68.62:40641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:47.493544 22346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:47.494652 22346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:47.503976 22352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:47.504031 22346 server_base.cc:1034] running on GCE node
W20260927 10:53:47.503983 22354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:47.503996 22351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:47.504590 22346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:47.505160 22346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:47.506326 22346 hybrid_clock.cc:648] HybridClock initialized: now 1790506427506309 us; error 27 us; skew 500 ppm
Sep 27 10:53:47 dist-test-slave-tz51 krb5kdc[22089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506427, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:47.509111 22346 init.cc:377] Logged in from keytab as oryx/127.14.68.2@KRBTEST.COM (short username oryx)
I20260927 10:53:47.509879 22346 webserver.cc:460] Webserver started at http://127.14.68.2:35155/ using document root <none> and password file <none>
I20260927 10:53:47.510373 22346 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:47.510428 22346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:47.510582 22346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:47.512524 22346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "77162430d03d4d87bbd367f0703a6042"
format_stamp: "Formatted at 2026-09-27 10:53:47 on dist-test-slave-tz51"
server_key: "51df0443f1a23dd5e96ef3cceaec27a7"
server_key_iv: "ed7b7d0259a900bcbe74fad163c69dd1"
server_key_version: "encryptionkey@0"
I20260927 10:53:47.513044 22346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "77162430d03d4d87bbd367f0703a6042"
format_stamp: "Formatted at 2026-09-27 10:53:47 on dist-test-slave-tz51"
server_key: "51df0443f1a23dd5e96ef3cceaec27a7"
server_key_iv: "ed7b7d0259a900bcbe74fad163c69dd1"
server_key_version: "encryptionkey@0"
I20260927 10:53:47.516427 22346 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:53:47.518612 22361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:47.519836 22346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:47.519956 22346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "77162430d03d4d87bbd367f0703a6042"
format_stamp: "Formatted at 2026-09-27 10:53:47 on dist-test-slave-tz51"
server_key: "51df0443f1a23dd5e96ef3cceaec27a7"
server_key_iv: "ed7b7d0259a900bcbe74fad163c69dd1"
server_key_version: "encryptionkey@0"
I20260927 10:53:47.520057 22346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:47.536360 22346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:47.538893 22346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:47.539080 22346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:47.540398 22346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:47.540472 22346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:47.540537 22346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:47.540571 22346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:53:47 dist-test-slave-tz51 krb5kdc[22089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506427, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM
I20260927 10:53:47.551214 22346 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:33069
I20260927 10:53:47.551270 22476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:33069 every 8 connection(s)
I20260927 10:53:47.552326 22346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:53:47.553285 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 22346
I20260927 10:53:47.553429 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:53:47.553707 14608 external_mini_cluster.cc:1428] Setting key 7bf52e69db8817ffc344d9e6c0c60d8d
I20260927 10:53:47.554018 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.541218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:50127 (local address 127.14.68.62:40641)
0927 10:53:47.541391 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.541398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.542160 (+ 762us) server_negotiation.cc:407] Connection header received
0927 10:53:47.543277 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.543283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.543350 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.543458 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:47.544862 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.545397 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.546787 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.547016 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.547143 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.550014 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:47.550021 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:47.550026 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:47.550059 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:47.551774 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:47.552243 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:47.552248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:47.552250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:47.552339 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:47.552754 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:47.552757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:47.552759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:47.552991 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:47.553086 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.553719 (+ 633us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.553848 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:53:47.554912 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.541572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40641 (local address 127.14.68.2:50127)
0927 10:53:47.542001 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:47.542048 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:47.543054 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:47.543638 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:47.543653 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:47.544137 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:47.544701 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.544724 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.545522 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.545525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.546652 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.546686 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.547200 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.547205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.547371 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.547973 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:47.547991 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:47.549836 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:47.551876 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:47.551880 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:47.551891 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:47.552133 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:47.552463 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:47.552467 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:47.552469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:47.552642 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:47.553137 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:47.553143 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:47.553447 (+ 304us) client_negotiation.cc:769] Sending connection context
0927 10:53:47.553683 (+ 236us) client_negotiation.cc:240] Negotiation successful
0927 10:53:47.553984 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":121,"threads_started":1}
I20260927 10:53:47.556291 22477 heartbeater.cc:347] Connected to a master server at 127.14.68.62:40641
I20260927 10:53:47.556418 22477 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:47.556938 22477 heartbeater.cc:511] Master 127.14.68.62:40641 requested a full tablet report, sending...
I20260927 10:53:47.557941 22133 ts_manager.cc:194] Registered new tserver with Master: 77162430d03d4d87bbd367f0703a6042 (127.14.68.2:33069)
I20260927 10:53:47.558740 22133 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:50127
WARNING: no policy specified for oryx/127.14.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.68.3@KRBTEST.COM" created.
I20260927 10:53:47.569883 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.558526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:40121 (local address 127.14.68.62:40641)
0927 10:53:47.558634 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.558638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.563233 (+ 4595us) server_negotiation.cc:407] Connection header received
0927 10:53:47.563344 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.563346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.563384 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.563450 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:47.564002 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.564554 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.565329 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.565616 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.565734 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.566800 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:47.566804 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:47.566806 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:47.566829 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:47.568271 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:47.568709 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:47.568714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:47.568716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:47.568792 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:47.569062 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:47.569066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:47.569068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:47.569271 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:47.569378 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.569629 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.569734 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 10:53:47.569909 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.558466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40641 (local address 127.14.68.2:40121)
0927 10:53:47.563141 (+ 4675us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:47.563159 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:47.563262 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:47.563481 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:47.563483 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:47.563641 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:47.563799 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.563805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.564669 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.564672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.565214 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.565222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.565728 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.565732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.565821 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.566358 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:47.566369 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:47.566658 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:47.568392 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:47.568395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:47.568398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:47.568605 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:47.568882 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:47.568887 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:47.568889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:47.568972 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:47.569369 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:47.569372 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:47.569470 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 10:53:47.569576 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 10:53:47.569722 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4570,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.3.keytab.
Entry for principal oryx/127.14.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.3.keytab.
I20260927 10:53:47.580404 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:40641
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.3.keytab
--principal=oryx/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:47.690119 22486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:47.690372 22486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:47.690416 22486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:47.694546 22486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:47.694600 22486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:47.694676 22486 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:53:47.694732 22486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:53:47.699805 22486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.3.keytab
--principal=oryx/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.68.62:40641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:47.700815 22486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:47.701805 22486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:47.711118 22491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:47.711104 22492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:47.711104 22494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:47.711580 22486 server_base.cc:1034] running on GCE node
I20260927 10:53:47.711967 22486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:47.712587 22486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:47.713768 22486 hybrid_clock.cc:648] HybridClock initialized: now 1790506427713756 us; error 34 us; skew 500 ppm
Sep 27 10:53:47 dist-test-slave-tz51 krb5kdc[22089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506427, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:47.716490 22486 init.cc:377] Logged in from keytab as oryx/127.14.68.3@KRBTEST.COM (short username oryx)
I20260927 10:53:47.717321 22486 webserver.cc:460] Webserver started at http://127.14.68.3:38099/ using document root <none> and password file <none>
I20260927 10:53:47.717857 22486 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:47.717941 22486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:47.718165 22486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:47.720172 22486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "cde272c487324aa1ac82c3e2bc74267f"
format_stamp: "Formatted at 2026-09-27 10:53:47 on dist-test-slave-tz51"
server_key: "d84fc1b62f0e1b7f8c8b44d8827def33"
server_key_iv: "d15ce7d498cab5a74de08c099535a095"
server_key_version: "encryptionkey@0"
I20260927 10:53:47.720714 22486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "cde272c487324aa1ac82c3e2bc74267f"
format_stamp: "Formatted at 2026-09-27 10:53:47 on dist-test-slave-tz51"
server_key: "d84fc1b62f0e1b7f8c8b44d8827def33"
server_key_iv: "d15ce7d498cab5a74de08c099535a095"
server_key_version: "encryptionkey@0"
I20260927 10:53:47.724433 22486 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:53:47.726805 22501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:47.728032 22486 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:47.728170 22486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "cde272c487324aa1ac82c3e2bc74267f"
format_stamp: "Formatted at 2026-09-27 10:53:47 on dist-test-slave-tz51"
server_key: "d84fc1b62f0e1b7f8c8b44d8827def33"
server_key_iv: "d15ce7d498cab5a74de08c099535a095"
server_key_version: "encryptionkey@0"
I20260927 10:53:47.728281 22486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:47.740134 22486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:47.742676 22486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:47.742862 22486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:47.744320 22486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:47.744390 22486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:47.744458 22486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:47.744510 22486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:53:47 dist-test-slave-tz51 krb5kdc[22089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506427, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM
I20260927 10:53:47.755353 22486 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:43703
I20260927 10:53:47.755460 22616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:43703 every 8 connection(s)
I20260927 10:53:47.756508 22486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:53:47.758437 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.744955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:53849 (local address 127.14.68.62:40641)
0927 10:53:47.745098 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.745103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.745933 (+ 830us) server_negotiation.cc:407] Connection header received
0927 10:53:47.746870 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.746873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.746937 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.747047 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:47.748727 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.749326 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.750599 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.750823 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.750986 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.754595 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:47.754599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:47.754602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:47.754638 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:47.756383 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:47.756972 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:47.756975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:47.756977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:47.757065 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:47.757466 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:47.757468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:47.757469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:47.757714 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:47.757795 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.758219 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.758314 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:53:47.759220 22510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.745363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40641 (local address 127.14.68.3:53849)
0927 10:53:47.745796 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:47.745843 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:47.746675 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:47.747366 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:47.747378 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:47.747918 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:47.748572 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.748597 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.749452 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.749457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.750466 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.750476 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.751037 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.751041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.751244 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.751871 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:47.751897 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:47.754430 (+ 2533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:47.756518 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:47.756523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:47.756537 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:47.756843 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:47.757189 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:47.757194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:47.757196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:47.757359 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:47.757808 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:47.757812 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:47.758017 (+ 205us) client_negotiation.cc:769] Sending connection context
0927 10:53:47.758180 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 10:53:47.758418 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":121,"threads_started":1}
I20260927 10:53:47.760483 22617 heartbeater.cc:347] Connected to a master server at 127.14.68.62:40641
I20260927 10:53:47.760625 22617 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:47.761113 22617 heartbeater.cc:511] Master 127.14.68.62:40641 requested a full tablet report, sending...
I20260927 10:53:47.762123 22134 ts_manager.cc:194] Registered new tserver with Master: cde272c487324aa1ac82c3e2bc74267f (127.14.68.3:43703)
I20260927 10:53:47.762900 22134 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:53849
I20260927 10:53:47.766256 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 22486
I20260927 10:53:47.766350 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:53:47.766741 14608 external_mini_cluster.cc:1428] Setting key f265eb9c05243155a6a16ef2a857c519
I20260927 10:53:47.768957 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:53:47.769961 22623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.762564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40641 (local address 127.14.68.3:59073)
0927 10:53:47.762850 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:47.762864 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:47.762965 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:47.763279 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:47.763282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:47.763480 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:47.763587 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.763592 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.764557 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.764560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.765098 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.765105 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.765589 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.765591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.765660 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.766165 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:47.766176 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:47.766594 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:47.768525 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:47.768528 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:47.768530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:47.768791 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:47.769087 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:47.769090 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:47.769092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:47.769190 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:47.769585 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:47.769587 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:47.769654 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 10:53:47.769732 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 10:53:47.769827 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":123,"threads_started":1}
I20260927 10:53:47.770179 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.762690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:59073 (local address 127.14.68.62:40641)
0927 10:53:47.762805 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.762809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.762937 (+ 128us) server_negotiation.cc:407] Connection header received
0927 10:53:47.763101 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.763104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.763172 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.763250 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:47.763726 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.764440 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.765208 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.765494 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.765573 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.766744 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:47.766749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:47.766756 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:47.766784 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:47.768421 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:47.768873 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:47.768893 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:47.768897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:47.768985 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:47.769279 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:47.769282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:47.769284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:47.769493 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:47.769593 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.769818 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.769941 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 10:53:47 dist-test-slave-tz51 krb5kdc[22089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506427, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:53:47 dist-test-slave-tz51 krb5kdc[22089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:47 dist-test-slave-tz51 krb5kdc[22089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 10:53:47.792502 22183 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:47.786563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53714 (local address 127.14.68.62:40641)
0927 10:53:47.786698 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.786701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.786711 (+ 10us) server_negotiation.cc:407] Connection header received
0927 10:53:47.786744 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.786746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.786780 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.786870 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:47.787569 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.788138 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.788993 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.789211 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.789319 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.792339 (+ 3020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53714: BlockingRecv error: recv got EOF from 127.0.0.1:53714 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 10:53:47 dist-test-slave-tz51 krb5kdc[22089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506427, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM
I20260927 10:53:47.801612 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.793982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53722 (local address 127.14.68.62:40641)
0927 10:53:47.794115 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.794119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.794295 (+ 176us) server_negotiation.cc:407] Connection header received
0927 10:53:47.794381 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.794383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.794419 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.794480 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:47.795122 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.795600 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.796389 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.796624 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.796732 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.799042 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:47.799045 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:47.799047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:47.799105 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:47.800326 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:47.800693 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:47.800695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:47.800696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:47.800764 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:47.800980 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:47.800982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:47.800983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:47.801150 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:47.801243 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.801425 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.801503 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:53:47.804260 22134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:53:47.807103 22134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:47.821116 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.815084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:43703 (local address 127.0.0.1:48630)
0927 10:53:47.815545 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:47.815590 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:47.815738 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:47.816579 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:47.816584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:47.816608 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:47.816753 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.816760 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.818100 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.818106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.819440 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.819450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.820559 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.820563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.820674 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.820718 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 10:53:47.820837 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 10:53:47.820956 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":74,"threads_started":1}
I20260927 10:53:47.821179 22641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.815234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48630 (local address 127.14.68.3:43703)
0927 10:53:47.816044 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.816062 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.816085 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:53:47.816178 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.816186 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.816397 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.816906 (+ 509us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:47.817042 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.817997 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.819577 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.820430 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.820543 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.820706 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.820884 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.820984 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":112,"threads_started":1}
I20260927 10:53:47.821475 22643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.816061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53596 (local address 127.14.68.2:33069)
0927 10:53:47.816987 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.816998 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.817020 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:53:47.817093 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.817098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.817266 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.817407 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:47.817721 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.818685 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.819876 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.820451 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.820573 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.820733 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.821201 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.821309 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":829,"thread_start_us":106,"threads_started":1}
I20260927 10:53:47.821692 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.816368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:33069 (local address 127.0.0.1:53596)
0927 10:53:47.816642 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:47.816655 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:47.816747 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:47.817493 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:47.817496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:47.817515 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:47.817589 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.817594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.818789 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.818794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.819745 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.819757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.820796 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.820801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.820938 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.820958 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:53:47.821023 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 10:53:47.821079 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":25,"thread_start_us":82,"threads_started":1}
I20260927 10:53:47.821756 22642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.815908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58002 (local address 127.14.68.1:35727)
0927 10:53:47.816208 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.816221 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.816366 (+ 145us) server_negotiation.cc:407] Connection header received
0927 10:53:47.816500 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.816508 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.816708 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.817120 (+ 412us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:47.817268 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.818335 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.819683 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.820430 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.820543 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.820706 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.821495 (+ 789us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.821607 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":118,"threads_started":1}
I20260927 10:53:47.821954 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.815156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:35727 (local address 127.0.0.1:58002)
0927 10:53:47.816264 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:47.816277 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:47.816392 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:47.816859 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:47.816862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:47.816876 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:47.816975 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.816983 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.818453 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.818457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.819565 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.819573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.820788 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.820791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.820980 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.821004 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 10:53:47.821078 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 10:53:47.821152 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1044,"mutex_wait_us":48,"thread_start_us":93,"threads_started":1}
I20260927 10:53:47.824360 22411 tablet_service.cc:1467] Processing CreateTablet for tablet b9b2d4d684854c9bb98be2315cd54b1b (DEFAULT_TABLE table=test-table [id=ca2365803d794b58879dd296303e4786]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:47.824313 22551 tablet_service.cc:1467] Processing CreateTablet for tablet b9b2d4d684854c9bb98be2315cd54b1b (DEFAULT_TABLE table=test-table [id=ca2365803d794b58879dd296303e4786]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:47.825251 22551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b2d4d684854c9bb98be2315cd54b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:47.825250 22411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b2d4d684854c9bb98be2315cd54b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:47.832212 22645 tablet_bootstrap.cc:492] T b9b2d4d684854c9bb98be2315cd54b1b P 77162430d03d4d87bbd367f0703a6042: Bootstrap starting.
I20260927 10:53:47.833149 22271 tablet_service.cc:1467] Processing CreateTablet for tablet b9b2d4d684854c9bb98be2315cd54b1b (DEFAULT_TABLE table=test-table [id=ca2365803d794b58879dd296303e4786]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:47.834034 22271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b2d4d684854c9bb98be2315cd54b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:47.834182 22645 tablet_bootstrap.cc:654] T b9b2d4d684854c9bb98be2315cd54b1b P 77162430d03d4d87bbd367f0703a6042: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:47.834880 22645 log.cc:824] T b9b2d4d684854c9bb98be2315cd54b1b P 77162430d03d4d87bbd367f0703a6042: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:47.835475 22646 tablet_bootstrap.cc:492] T b9b2d4d684854c9bb98be2315cd54b1b P cde272c487324aa1ac82c3e2bc74267f: Bootstrap starting.
I20260927 10:53:47.836887 22645 tablet_bootstrap.cc:492] T b9b2d4d684854c9bb98be2315cd54b1b P 77162430d03d4d87bbd367f0703a6042: No bootstrap required, opened a new log
I20260927 10:53:47.837083 22645 ts_tablet_manager.cc:1397] T b9b2d4d684854c9bb98be2315cd54b1b P 77162430d03d4d87bbd367f0703a6042: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:53:47.837313 22646 tablet_bootstrap.cc:654] T b9b2d4d684854c9bb98be2315cd54b1b P cde272c487324aa1ac82c3e2bc74267f: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:47.838053 22646 log.cc:824] T b9b2d4d684854c9bb98be2315cd54b1b P cde272c487324aa1ac82c3e2bc74267f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:47.839913 22646 tablet_bootstrap.cc:492] T b9b2d4d684854c9bb98be2315cd54b1b P cde272c487324aa1ac82c3e2bc74267f: No bootstrap required, opened a new log
I20260927 10:53:47.840109 22646 ts_tablet_manager.cc:1397] T b9b2d4d684854c9bb98be2315cd54b1b P cde272c487324aa1ac82c3e2bc74267f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:53:47.841159 22645 raft_consensus.cc:348] T b9b2d4d684854c9bb98be2315cd54b1b P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } } peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } }
I20260927 10:53:47.841437 22645 raft_consensus.cc:374] T b9b2d4d684854c9bb98be2315cd54b1b P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:47.841492 22645 raft_consensus.cc:729] T b9b2d4d684854c9bb98be2315cd54b1b P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77162430d03d4d87bbd367f0703a6042, State: Initialized, Role: FOLLOWER
I20260927 10:53:47.841763 22649 tablet_bootstrap.cc:492] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a: Bootstrap starting.
I20260927 10:53:47.841966 22645 consensus_queue.cc:260] T b9b2d4d684854c9bb98be2315cd54b1b P 77162430d03d4d87bbd367f0703a6042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } } peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } }
I20260927 10:53:47.842612 22477 heartbeater.cc:503] Master 127.14.68.62:40641 was elected leader, sending a full tablet report...
I20260927 10:53:47.842973 22645 ts_tablet_manager.cc:1428] T b9b2d4d684854c9bb98be2315cd54b1b P 77162430d03d4d87bbd367f0703a6042: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:53:47.843673 22646 raft_consensus.cc:348] T b9b2d4d684854c9bb98be2315cd54b1b P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } } peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } }
I20260927 10:53:47.843952 22646 raft_consensus.cc:374] T b9b2d4d684854c9bb98be2315cd54b1b P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:47.844012 22646 raft_consensus.cc:729] T b9b2d4d684854c9bb98be2315cd54b1b P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde272c487324aa1ac82c3e2bc74267f, State: Initialized, Role: FOLLOWER
I20260927 10:53:47.844612 22649 tablet_bootstrap.cc:654] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:47.844787 22646 consensus_queue.cc:260] T b9b2d4d684854c9bb98be2315cd54b1b P cde272c487324aa1ac82c3e2bc74267f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } } peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } }
I20260927 10:53:47.845494 22617 heartbeater.cc:503] Master 127.14.68.62:40641 was elected leader, sending a full tablet report...
I20260927 10:53:47.845475 22649 log.cc:824] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:47.845806 22646 ts_tablet_manager.cc:1428] T b9b2d4d684854c9bb98be2315cd54b1b P cde272c487324aa1ac82c3e2bc74267f: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260927 10:53:47.847572 22649 tablet_bootstrap.cc:492] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a: No bootstrap required, opened a new log
I20260927 10:53:47.847788 22649 ts_tablet_manager.cc:1397] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:53:47.850808 22649 raft_consensus.cc:348] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } } peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } }
I20260927 10:53:47.851045 22649 raft_consensus.cc:374] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:47.851107 22649 raft_consensus.cc:729] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5729ed1c02e7442fb08b248c96e9f55a, State: Initialized, Role: FOLLOWER
I20260927 10:53:47.851475 22649 consensus_queue.cc:260] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } } peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } }
I20260927 10:53:47.852128 22337 heartbeater.cc:503] Master 127.14.68.62:40641 was elected leader, sending a full tablet report...
I20260927 10:53:47.852495 22649 ts_tablet_manager.cc:1428] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260927 10:53:47.886792 22653 raft_consensus.cc:482] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:47.887012 22653 raft_consensus.cc:504] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } } peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } }
I20260927 10:53:47.888266 22653 leader_election.cc:288] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cde272c487324aa1ac82c3e2bc74267f (127.14.68.3:43703), 77162430d03d4d87bbd367f0703a6042 (127.14.68.2:33069)
I20260927 10:53:47.892577 22643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.888684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:45615 (local address 127.14.68.2:33069)
0927 10:53:47.888804 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.888810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.888917 (+ 107us) server_negotiation.cc:407] Connection header received
0927 10:53:47.889051 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.889054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.889100 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.889183 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:47.889514 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.890030 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.891340 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.891835 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.891921 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.892000 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.892339 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.892424 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:53:47.892603 22641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.888557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:56709 (local address 127.14.68.3:43703)
0927 10:53:47.888686 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.888691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.888878 (+ 187us) server_negotiation.cc:407] Connection header received
0927 10:53:47.889055 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.889057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.889100 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.889183 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:47.889571 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.890068 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.891420 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.891897 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.892005 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.892072 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.892380 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.892461 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:53:47.892622 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.888436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:43703 (local address 127.14.68.1:56709)
0927 10:53:47.888766 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:47.888781 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:47.888919 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:47.889197 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:47.889199 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:47.889242 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:47.889401 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.889406 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.890207 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.890210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.891271 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.891301 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.892019 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.892023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.892209 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.892254 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 10:53:47.892345 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 10:53:47.892447 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":165,"spinlock_wait_cycles":32384,"thread_start_us":98,"threads_started":1}
I20260927 10:53:47.892629 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.888521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:33069 (local address 127.14.68.1:45615)
0927 10:53:47.888806 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:47.888818 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:47.888940 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:47.889198 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:47.889201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:47.889243 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:47.889390 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.889395 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.890140 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.890147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.891215 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:47.891225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.891952 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.891958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:47.892103 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.892173 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 10:53:47.892321 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 10:53:47.892416 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":87,"spinlock_wait_cycles":9344,"thread_start_us":84,"threads_started":1}
I20260927 10:53:47.893224 22571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b2d4d684854c9bb98be2315cd54b1b" candidate_uuid: "5729ed1c02e7442fb08b248c96e9f55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde272c487324aa1ac82c3e2bc74267f" is_pre_election: true
I20260927 10:53:47.893236 22431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b2d4d684854c9bb98be2315cd54b1b" candidate_uuid: "5729ed1c02e7442fb08b248c96e9f55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77162430d03d4d87bbd367f0703a6042" is_pre_election: true
I20260927 10:53:47.893550 22571 raft_consensus.cc:2454] T b9b2d4d684854c9bb98be2315cd54b1b P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5729ed1c02e7442fb08b248c96e9f55a in term 0.
I20260927 10:53:47.893534 22431 raft_consensus.cc:2454] T b9b2d4d684854c9bb98be2315cd54b1b P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5729ed1c02e7442fb08b248c96e9f55a in term 0.
I20260927 10:53:47.894074 22225 leader_election.cc:302] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5729ed1c02e7442fb08b248c96e9f55a, 77162430d03d4d87bbd367f0703a6042, cde272c487324aa1ac82c3e2bc74267f; no voters:
I20260927 10:53:47.894343 22653 raft_consensus.cc:2781] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:47.894418 22653 raft_consensus.cc:482] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:47.894447 22653 raft_consensus.cc:3037] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:47.895496 22653 raft_consensus.cc:504] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } } peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } }
I20260927 10:53:47.895856 22653 leader_election.cc:288] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a [CANDIDATE]: Term 1 election: Requested vote from peers cde272c487324aa1ac82c3e2bc74267f (127.14.68.3:43703), 77162430d03d4d87bbd367f0703a6042 (127.14.68.2:33069)
I20260927 10:53:47.896196 22431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b2d4d684854c9bb98be2315cd54b1b" candidate_uuid: "5729ed1c02e7442fb08b248c96e9f55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77162430d03d4d87bbd367f0703a6042"
I20260927 10:53:47.896193 22571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b2d4d684854c9bb98be2315cd54b1b" candidate_uuid: "5729ed1c02e7442fb08b248c96e9f55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde272c487324aa1ac82c3e2bc74267f"
I20260927 10:53:47.896320 22571 raft_consensus.cc:3037] T b9b2d4d684854c9bb98be2315cd54b1b P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:47.896320 22431 raft_consensus.cc:3037] T b9b2d4d684854c9bb98be2315cd54b1b P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:47.897606 22571 raft_consensus.cc:2454] T b9b2d4d684854c9bb98be2315cd54b1b P cde272c487324aa1ac82c3e2bc74267f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5729ed1c02e7442fb08b248c96e9f55a in term 1.
I20260927 10:53:47.897606 22431 raft_consensus.cc:2454] T b9b2d4d684854c9bb98be2315cd54b1b P 77162430d03d4d87bbd367f0703a6042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5729ed1c02e7442fb08b248c96e9f55a in term 1.
I20260927 10:53:47.897944 22223 leader_election.cc:302] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5729ed1c02e7442fb08b248c96e9f55a, 77162430d03d4d87bbd367f0703a6042; no voters:
I20260927 10:53:47.898166 22653 raft_consensus.cc:2781] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:47.898348 22653 raft_consensus.cc:686] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a [term 1 LEADER]: Becoming Leader. State: Replica: 5729ed1c02e7442fb08b248c96e9f55a, State: Running, Role: LEADER
I20260927 10:53:47.898680 22653 consensus_queue.cc:237] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } } peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } }
I20260927 10:53:47.902011 22133 catalog_manager.cc:5495] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5729ed1c02e7442fb08b248c96e9f55a (127.14.68.1). New cstate: current_term: 1 leader_uuid: "5729ed1c02e7442fb08b248c96e9f55a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } health_report { overall_health: UNKNOWN } } }
I20260927 10:53:47.919003 22642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:47.915596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58014 (local address 127.14.68.1:35727)
0927 10:53:47.915839 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:47.915843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:47.915855 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:53:47.915922 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:47.915924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:47.915977 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:47.916063 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:47.916297 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.916792 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.917680 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:47.917884 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:47.917958 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:47.918228 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:47.918660 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:47.918743 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:47.918843 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 10:53:47.918894 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260927 10:53:47.925043 22571 raft_consensus.cc:1261] T b9b2d4d684854c9bb98be2315cd54b1b P cde272c487324aa1ac82c3e2bc74267f [term 1 FOLLOWER]: Refusing update from remote peer 5729ed1c02e7442fb08b248c96e9f55a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:47.925472 22431 raft_consensus.cc:1261] T b9b2d4d684854c9bb98be2315cd54b1b P 77162430d03d4d87bbd367f0703a6042 [term 1 FOLLOWER]: Refusing update from remote peer 5729ed1c02e7442fb08b248c96e9f55a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:47.925742 22653 consensus_queue.cc:1035] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:47.926087 22656 consensus_queue.cc:1035] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a [LEADER]: Connected to new peer: Peer: permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:47.934953 22661 mvcc.cc:205] Tried to move back new op lower bound from 7333914328773275648 to 7333914328675540992. Current Snapshot: MvccSnapshot[applied={T|T < 7333914328773275648}]
I20260927 10:53:47.943152 22134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:47.943347 22134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:47.945852 22134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc: Sending DeleteTablet for 3 replicas of tablet b9b2d4d684854c9bb98be2315cd54b1b
I20260927 10:53:47.946704 22551 tablet_service.cc:1514] Processing DeleteTablet for tablet b9b2d4d684854c9bb98be2315cd54b1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:47 UTC) from {username='oryx', principal='oryx/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:48630
I20260927 10:53:47.946753 22411 tablet_service.cc:1514] Processing DeleteTablet for tablet b9b2d4d684854c9bb98be2315cd54b1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:47 UTC) from {username='oryx', principal='oryx/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:53596
I20260927 10:53:47.947247 22669 tablet_replica.cc:330] T b9b2d4d684854c9bb98be2315cd54b1b P cde272c487324aa1ac82c3e2bc74267f: stopping tablet replica
I20260927 10:53:47.947233 22271 tablet_service.cc:1514] Processing DeleteTablet for tablet b9b2d4d684854c9bb98be2315cd54b1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:47 UTC) from {username='oryx', principal='oryx/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:58002
I20260927 10:53:47.947582 22669 raft_consensus.cc:2229] T b9b2d4d684854c9bb98be2315cd54b1b P cde272c487324aa1ac82c3e2bc74267f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:47.947737 22671 tablet_replica.cc:330] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a: stopping tablet replica
I20260927 10:53:47.948060 22671 raft_consensus.cc:2229] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:53:47.948069 22670 tablet_replica.cc:330] T b9b2d4d684854c9bb98be2315cd54b1b P 77162430d03d4d87bbd367f0703a6042: stopping tablet replica
I20260927 10:53:47.948346 22670 raft_consensus.cc:2229] T b9b2d4d684854c9bb98be2315cd54b1b P 77162430d03d4d87bbd367f0703a6042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:47.948287 22134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 10:53:47.948661 22670 raft_consensus.cc:2258] T b9b2d4d684854c9bb98be2315cd54b1b P 77162430d03d4d87bbd367f0703a6042 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 10:53:47.948685 22134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:47.948782 22671 raft_consensus.cc:2258] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:47.949417 22669 raft_consensus.cc:2258] T b9b2d4d684854c9bb98be2315cd54b1b P cde272c487324aa1ac82c3e2bc74267f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:47.950258 22670 ts_tablet_manager.cc:1905] T b9b2d4d684854c9bb98be2315cd54b1b P 77162430d03d4d87bbd367f0703a6042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:47.951100 22671 ts_tablet_manager.cc:1905] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:47.953444 22670 ts_tablet_manager.cc:1918] T b9b2d4d684854c9bb98be2315cd54b1b P 77162430d03d4d87bbd367f0703a6042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:47.953550 22670 log.cc:1196] T b9b2d4d684854c9bb98be2315cd54b1b P 77162430d03d4d87bbd367f0703a6042: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/b9b2d4d684854c9bb98be2315cd54b1b
I20260927 10:53:47.953855 22670 ts_tablet_manager.cc:1939] T b9b2d4d684854c9bb98be2315cd54b1b P 77162430d03d4d87bbd367f0703a6042: Deleting consensus metadata
I20260927 10:53:47.954280 22551 tablet_service.cc:1467] Processing CreateTablet for tablet 2afdc0a5c7c24b08ba972a8e2c3ea31b (DEFAULT_TABLE table=test-table [id=30d9c3f5ecad4aec91e0f2c8bbe20a06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:47.954339 22671 ts_tablet_manager.cc:1918] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:47.954536 22671 log.cc:1196] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-0/wal/wals/b9b2d4d684854c9bb98be2315cd54b1b
I20260927 10:53:47.954558 22551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2afdc0a5c7c24b08ba972a8e2c3ea31b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:47.954939 22671 ts_tablet_manager.cc:1939] T b9b2d4d684854c9bb98be2315cd54b1b P 5729ed1c02e7442fb08b248c96e9f55a: Deleting consensus metadata
I20260927 10:53:47.955291 22119 catalog_manager.cc:4841] TS 77162430d03d4d87bbd367f0703a6042 (127.14.68.2:33069): tablet b9b2d4d684854c9bb98be2315cd54b1b (table test-table [id=ca2365803d794b58879dd296303e4786]) successfully deleted
I20260927 10:53:47.955777 22271 tablet_service.cc:1467] Processing CreateTablet for tablet 2afdc0a5c7c24b08ba972a8e2c3ea31b (DEFAULT_TABLE table=test-table [id=30d9c3f5ecad4aec91e0f2c8bbe20a06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:47.956013 22271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2afdc0a5c7c24b08ba972a8e2c3ea31b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:47.956224 22119 catalog_manager.cc:4841] TS 5729ed1c02e7442fb08b248c96e9f55a (127.14.68.1:35727): tablet b9b2d4d684854c9bb98be2315cd54b1b (table test-table [id=ca2365803d794b58879dd296303e4786]) successfully deleted
I20260927 10:53:47.957065 22646 tablet_bootstrap.cc:492] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P cde272c487324aa1ac82c3e2bc74267f: Bootstrap starting.
I20260927 10:53:47.957909 22646 tablet_bootstrap.cc:654] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P cde272c487324aa1ac82c3e2bc74267f: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:47.958073 22649 tablet_bootstrap.cc:492] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a: Bootstrap starting.
I20260927 10:53:47.958391 22411 tablet_service.cc:1467] Processing CreateTablet for tablet 2afdc0a5c7c24b08ba972a8e2c3ea31b (DEFAULT_TABLE table=test-table [id=30d9c3f5ecad4aec91e0f2c8bbe20a06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:47.958621 22411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2afdc0a5c7c24b08ba972a8e2c3ea31b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:47.959019 22649 tablet_bootstrap.cc:654] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:47.960148 22649 tablet_bootstrap.cc:492] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a: No bootstrap required, opened a new log
I20260927 10:53:47.960148 22646 tablet_bootstrap.cc:492] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P cde272c487324aa1ac82c3e2bc74267f: No bootstrap required, opened a new log
I20260927 10:53:47.960228 22649 ts_tablet_manager.cc:1397] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 10:53:47.960228 22646 ts_tablet_manager.cc:1397] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P cde272c487324aa1ac82c3e2bc74267f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 10:53:47.960677 22646 raft_consensus.cc:348] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:47.960685 22649 raft_consensus.cc:348] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:47.960784 22646 raft_consensus.cc:374] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:47.960790 22645 tablet_bootstrap.cc:492] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 77162430d03d4d87bbd367f0703a6042: Bootstrap starting.
I20260927 10:53:47.960798 22649 raft_consensus.cc:374] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:47.960824 22646 raft_consensus.cc:729] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde272c487324aa1ac82c3e2bc74267f, State: Initialized, Role: FOLLOWER
I20260927 10:53:47.960839 22649 raft_consensus.cc:729] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5729ed1c02e7442fb08b248c96e9f55a, State: Initialized, Role: FOLLOWER
I20260927 10:53:47.960947 22646 consensus_queue.cc:260] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P cde272c487324aa1ac82c3e2bc74267f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:47.960947 22649 consensus_queue.cc:260] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:47.961186 22649 ts_tablet_manager.cc:1428] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:53:47.961205 22646 ts_tablet_manager.cc:1428] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P cde272c487324aa1ac82c3e2bc74267f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:53:47.961669 22645 tablet_bootstrap.cc:654] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 77162430d03d4d87bbd367f0703a6042: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:47.963240 22645 tablet_bootstrap.cc:492] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 77162430d03d4d87bbd367f0703a6042: No bootstrap required, opened a new log
I20260927 10:53:47.963317 22645 ts_tablet_manager.cc:1397] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 77162430d03d4d87bbd367f0703a6042: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:53:47.963761 22645 raft_consensus.cc:348] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:47.963867 22645 raft_consensus.cc:374] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:47.963896 22645 raft_consensus.cc:729] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77162430d03d4d87bbd367f0703a6042, State: Initialized, Role: FOLLOWER
I20260927 10:53:47.964005 22645 consensus_queue.cc:260] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 77162430d03d4d87bbd367f0703a6042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:47.964236 22645 ts_tablet_manager.cc:1428] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 77162430d03d4d87bbd367f0703a6042: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:53:47.966446 22669 ts_tablet_manager.cc:1905] T b9b2d4d684854c9bb98be2315cd54b1b P cde272c487324aa1ac82c3e2bc74267f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:47.969004 22669 ts_tablet_manager.cc:1918] T b9b2d4d684854c9bb98be2315cd54b1b P cde272c487324aa1ac82c3e2bc74267f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:47.969097 22669 log.cc:1196] T b9b2d4d684854c9bb98be2315cd54b1b P cde272c487324aa1ac82c3e2bc74267f: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/b9b2d4d684854c9bb98be2315cd54b1b
I20260927 10:53:47.969401 22669 ts_tablet_manager.cc:1939] T b9b2d4d684854c9bb98be2315cd54b1b P cde272c487324aa1ac82c3e2bc74267f: Deleting consensus metadata
I20260927 10:53:47.970587 22121 catalog_manager.cc:4841] TS cde272c487324aa1ac82c3e2bc74267f (127.14.68.3:43703): tablet b9b2d4d684854c9bb98be2315cd54b1b (table test-table [id=ca2365803d794b58879dd296303e4786]) successfully deleted
W20260927 10:53:48.007462 22618 tablet.cc:2369] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P cde272c487324aa1ac82c3e2bc74267f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:53:48.054598 22478 tablet.cc:2369] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 77162430d03d4d87bbd367f0703a6042: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:53:48.100611 22338 tablet.cc:2369] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:53:48.124249 22664 raft_consensus.cc:482] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:48.124406 22664 raft_consensus.cc:504] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.124847 22664 leader_election.cc:288] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77162430d03d4d87bbd367f0703a6042 (127.14.68.2:33069), cde272c487324aa1ac82c3e2bc74267f (127.14.68.3:43703)
I20260927 10:53:48.125283 22431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2afdc0a5c7c24b08ba972a8e2c3ea31b" candidate_uuid: "5729ed1c02e7442fb08b248c96e9f55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77162430d03d4d87bbd367f0703a6042" is_pre_election: true
I20260927 10:53:48.125296 22571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2afdc0a5c7c24b08ba972a8e2c3ea31b" candidate_uuid: "5729ed1c02e7442fb08b248c96e9f55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde272c487324aa1ac82c3e2bc74267f" is_pre_election: true
I20260927 10:53:48.125411 22431 raft_consensus.cc:2454] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5729ed1c02e7442fb08b248c96e9f55a in term 0.
I20260927 10:53:48.125469 22571 raft_consensus.cc:2454] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5729ed1c02e7442fb08b248c96e9f55a in term 0.
I20260927 10:53:48.125715 22223 leader_election.cc:302] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5729ed1c02e7442fb08b248c96e9f55a, 77162430d03d4d87bbd367f0703a6042; no voters:
I20260927 10:53:48.125880 22664 raft_consensus.cc:2781] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:48.125937 22664 raft_consensus.cc:482] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:48.125969 22664 raft_consensus.cc:3037] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:48.126947 22664 raft_consensus.cc:504] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.127267 22664 leader_election.cc:288] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a [CANDIDATE]: Term 1 election: Requested vote from peers 77162430d03d4d87bbd367f0703a6042 (127.14.68.2:33069), cde272c487324aa1ac82c3e2bc74267f (127.14.68.3:43703)
I20260927 10:53:48.127609 22431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2afdc0a5c7c24b08ba972a8e2c3ea31b" candidate_uuid: "5729ed1c02e7442fb08b248c96e9f55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77162430d03d4d87bbd367f0703a6042"
I20260927 10:53:48.127646 22571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2afdc0a5c7c24b08ba972a8e2c3ea31b" candidate_uuid: "5729ed1c02e7442fb08b248c96e9f55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde272c487324aa1ac82c3e2bc74267f"
I20260927 10:53:48.127758 22431 raft_consensus.cc:3037] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:48.127804 22571 raft_consensus.cc:3037] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:48.128612 22431 raft_consensus.cc:2454] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 77162430d03d4d87bbd367f0703a6042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5729ed1c02e7442fb08b248c96e9f55a in term 1.
I20260927 10:53:48.128705 22571 raft_consensus.cc:2454] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P cde272c487324aa1ac82c3e2bc74267f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5729ed1c02e7442fb08b248c96e9f55a in term 1.
I20260927 10:53:48.128922 22223 leader_election.cc:302] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5729ed1c02e7442fb08b248c96e9f55a, 77162430d03d4d87bbd367f0703a6042; no voters:
I20260927 10:53:48.129103 22664 raft_consensus.cc:2781] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:48.129222 22664 raft_consensus.cc:686] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a [term 1 LEADER]: Becoming Leader. State: Replica: 5729ed1c02e7442fb08b248c96e9f55a, State: Running, Role: LEADER
I20260927 10:53:48.129390 22664 consensus_queue.cc:237] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.131134 22132 catalog_manager.cc:5495] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5729ed1c02e7442fb08b248c96e9f55a (127.14.68.1). New cstate: current_term: 1 leader_uuid: "5729ed1c02e7442fb08b248c96e9f55a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } health_report { overall_health: UNKNOWN } } }
Sep 27 10:53:48 dist-test-slave-tz51 krb5kdc[22089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506427, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.62@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM
I20260927 10:53:48.188763 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:48.181275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40641 (local address 127.0.0.1:53738)
0927 10:53:48.181529 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:48.181541 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:48.181636 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:48.181849 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:48.181851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:48.182247 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:48.182335 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:48.182340 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.183006 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.183008 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:48.183536 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:48.183543 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.183960 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.183963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:48.184021 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.184607 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:48.184617 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:48.185931 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:48.187446 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:48.187454 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:48.187458 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:48.187671 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:48.187944 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:48.187946 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:48.187947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:48.188021 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:48.188325 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:48.188329 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:48.188419 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 10:53:48.188504 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 10:53:48.188618 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":77,"threads_started":1}
I20260927 10:53:48.188781 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:48.181370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53738 (local address 127.14.68.62:40641)
0927 10:53:48.181517 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:48.181524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:48.181611 (+ 87us) server_negotiation.cc:407] Connection header received
0927 10:53:48.181708 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:48.181711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:48.181750 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:48.181816 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:48.182427 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.182920 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.183636 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.183848 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.183946 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.186038 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:48.186041 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:48.186043 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:48.186079 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:48.187356 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:48.187801 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:48.187803 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:48.187804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:48.187870 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:48.188096 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:48.188098 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:48.188099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:48.188242 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:48.188311 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:48.188564 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 10:53:48.188671 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":60}
I20260927 10:53:48.192438 22132 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:53738:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 10:53:48.193171 22132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:48.197791 22271 tablet_service.cc:1467] Processing CreateTablet for tablet 7bd8d9e5ac044672a8a1d6848b8f6ee8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=099c45a0e9454a0ba413784995c41002]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:53:48.197897 22411 tablet_service.cc:1467] Processing CreateTablet for tablet 7bd8d9e5ac044672a8a1d6848b8f6ee8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=099c45a0e9454a0ba413784995c41002]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:53:48.198232 22271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bd8d9e5ac044672a8a1d6848b8f6ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:48.198349 22411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bd8d9e5ac044672a8a1d6848b8f6ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:48.200613 22649 tablet_bootstrap.cc:492] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 5729ed1c02e7442fb08b248c96e9f55a: Bootstrap starting.
I20260927 10:53:48.200700 22645 tablet_bootstrap.cc:492] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 77162430d03d4d87bbd367f0703a6042: Bootstrap starting.
I20260927 10:53:48.201315 22649 tablet_bootstrap.cc:654] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 5729ed1c02e7442fb08b248c96e9f55a: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:48.201426 22645 tablet_bootstrap.cc:654] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 77162430d03d4d87bbd367f0703a6042: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:48.202347 22649 tablet_bootstrap.cc:492] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 5729ed1c02e7442fb08b248c96e9f55a: No bootstrap required, opened a new log
I20260927 10:53:48.202427 22649 ts_tablet_manager.cc:1397] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 5729ed1c02e7442fb08b248c96e9f55a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 10:53:48.202600 22645 tablet_bootstrap.cc:492] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 77162430d03d4d87bbd367f0703a6042: No bootstrap required, opened a new log
I20260927 10:53:48.202656 22645 ts_tablet_manager.cc:1397] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 77162430d03d4d87bbd367f0703a6042: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:53:48.202550 22551 tablet_service.cc:1467] Processing CreateTablet for tablet 7bd8d9e5ac044672a8a1d6848b8f6ee8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=099c45a0e9454a0ba413784995c41002]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:53:48.202893 22551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bd8d9e5ac044672a8a1d6848b8f6ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:48.202848 22649 raft_consensus.cc:348] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.202950 22649 raft_consensus.cc:374] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:48.202978 22649 raft_consensus.cc:729] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5729ed1c02e7442fb08b248c96e9f55a, State: Initialized, Role: FOLLOWER
I20260927 10:53:48.202952 22645 raft_consensus.cc:348] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.203023 22645 raft_consensus.cc:374] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:48.203043 22645 raft_consensus.cc:729] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77162430d03d4d87bbd367f0703a6042, State: Initialized, Role: FOLLOWER
I20260927 10:53:48.203099 22649 consensus_queue.cc:260] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 5729ed1c02e7442fb08b248c96e9f55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.203205 22645 consensus_queue.cc:260] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 77162430d03d4d87bbd367f0703a6042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.203387 22649 ts_tablet_manager.cc:1428] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 5729ed1c02e7442fb08b248c96e9f55a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:53:48.203377 22664 tablet_replica.cc:439] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 5729ed1c02e7442fb08b248c96e9f55a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } } }
I20260927 10:53:48.203469 22645 ts_tablet_manager.cc:1428] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 77162430d03d4d87bbd367f0703a6042: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:53:48.203512 22664 tablet_replica.cc:442] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 5729ed1c02e7442fb08b248c96e9f55a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:53:48.204147 22650 tablet_replica.cc:439] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 77162430d03d4d87bbd367f0703a6042: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } } }
I20260927 10:53:48.204278 22650 tablet_replica.cc:442] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 77162430d03d4d87bbd367f0703a6042: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:53:48.205840 22646 tablet_bootstrap.cc:492] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f: Bootstrap starting.
I20260927 10:53:48.206857 22646 tablet_bootstrap.cc:654] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:48.207974 22646 tablet_bootstrap.cc:492] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f: No bootstrap required, opened a new log
I20260927 10:53:48.208076 22646 ts_tablet_manager.cc:1397] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 10:53:48.208472 22646 raft_consensus.cc:348] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.208582 22646 raft_consensus.cc:374] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:48.208627 22646 raft_consensus.cc:729] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde272c487324aa1ac82c3e2bc74267f, State: Initialized, Role: FOLLOWER
I20260927 10:53:48.208751 22646 consensus_queue.cc:260] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.208985 22646 ts_tablet_manager.cc:1428] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:53:48.209002 22651 tablet_replica.cc:439] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } } }
I20260927 10:53:48.209149 22651 tablet_replica.cc:442] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:53:48.247330 22651 raft_consensus.cc:482] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:48.247460 22651 raft_consensus.cc:504] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.248494 22651 leader_election.cc:288] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5729ed1c02e7442fb08b248c96e9f55a (127.14.68.1:35727), 77162430d03d4d87bbd367f0703a6042 (127.14.68.2:33069)
I20260927 10:53:48.252421 22642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:48.248871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:60131 (local address 127.14.68.1:35727)
0927 10:53:48.248999 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:48.249004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:48.249021 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:53:48.249082 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:48.249085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:48.249135 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:48.249223 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:48.249590 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.250094 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.251195 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.251849 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.251953 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.252020 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:48.252216 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 10:53:48.252289 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:53:48.252441 22510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:48.248730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:35727 (local address 127.14.68.3:60131)
0927 10:53:48.248872 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:48.248889 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:48.248979 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:48.249345 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:48.249348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:48.249366 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:48.249468 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:48.249473 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.250215 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.250222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:48.251035 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:48.251044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.251983 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.251987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:48.252083 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.252104 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:53:48.252173 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 10:53:48.252279 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":3}
I20260927 10:53:48.253330 22291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd8d9e5ac044672a8a1d6848b8f6ee8" candidate_uuid: "cde272c487324aa1ac82c3e2bc74267f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5729ed1c02e7442fb08b248c96e9f55a" is_pre_election: true
I20260927 10:53:48.253670 22685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:48.249022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:33069 (local address 127.14.68.3:43535)
0927 10:53:48.249397 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:48.249409 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:48.249529 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:48.249796 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:48.249799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:48.249814 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:48.250304 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:48.250311 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.251098 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.251101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:48.252384 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:48.252393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.253251 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.253255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:48.253357 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.253381 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 10:53:48.253463 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 10:53:48.253527 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":17,"thread_start_us":158,"threads_started":1}
I20260927 10:53:48.253630 22291 raft_consensus.cc:2454] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cde272c487324aa1ac82c3e2bc74267f in term 0.
I20260927 10:53:48.253690 22643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:48.249113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:43535 (local address 127.14.68.2:33069)
0927 10:53:48.249264 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:48.249269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:48.249497 (+ 228us) server_negotiation.cc:407] Connection header received
0927 10:53:48.249637 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:48.249641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:48.249697 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:48.249775 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:48.250421 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.250987 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.252526 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.253127 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.253240 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.253285 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:48.253503 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 10:53:48.253584 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:53:48.254153 22503 leader_election.cc:302] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5729ed1c02e7442fb08b248c96e9f55a, cde272c487324aa1ac82c3e2bc74267f; no voters:
I20260927 10:53:48.254207 22431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd8d9e5ac044672a8a1d6848b8f6ee8" candidate_uuid: "cde272c487324aa1ac82c3e2bc74267f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77162430d03d4d87bbd367f0703a6042" is_pre_election: true
I20260927 10:53:48.254334 22431 raft_consensus.cc:2454] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cde272c487324aa1ac82c3e2bc74267f in term 0.
I20260927 10:53:48.254426 22651 raft_consensus.cc:2781] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:48.254506 22651 raft_consensus.cc:482] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:48.254559 22651 raft_consensus.cc:3037] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:48.255647 22651 raft_consensus.cc:504] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.256053 22651 leader_election.cc:288] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f [CANDIDATE]: Term 1 election: Requested vote from peers 5729ed1c02e7442fb08b248c96e9f55a (127.14.68.1:35727), 77162430d03d4d87bbd367f0703a6042 (127.14.68.2:33069)
I20260927 10:53:48.256455 22291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd8d9e5ac044672a8a1d6848b8f6ee8" candidate_uuid: "cde272c487324aa1ac82c3e2bc74267f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5729ed1c02e7442fb08b248c96e9f55a"
I20260927 10:53:48.256488 22431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd8d9e5ac044672a8a1d6848b8f6ee8" candidate_uuid: "cde272c487324aa1ac82c3e2bc74267f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77162430d03d4d87bbd367f0703a6042"
I20260927 10:53:48.256592 22291 raft_consensus.cc:3037] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:48.256644 22431 raft_consensus.cc:3037] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:48.257516 22431 raft_consensus.cc:2454] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 77162430d03d4d87bbd367f0703a6042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cde272c487324aa1ac82c3e2bc74267f in term 1.
I20260927 10:53:48.257815 22503 leader_election.cc:302] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77162430d03d4d87bbd367f0703a6042, cde272c487324aa1ac82c3e2bc74267f; no voters:
I20260927 10:53:48.257830 22291 raft_consensus.cc:2454] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 5729ed1c02e7442fb08b248c96e9f55a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cde272c487324aa1ac82c3e2bc74267f in term 1.
I20260927 10:53:48.258004 22651 raft_consensus.cc:2781] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:48.258239 22651 raft_consensus.cc:686] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f [term 1 LEADER]: Becoming Leader. State: Replica: cde272c487324aa1ac82c3e2bc74267f, State: Running, Role: LEADER
I20260927 10:53:48.258484 22651 consensus_queue.cc:237] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.259531 22686 tablet_replica.cc:439] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f: TxnStatusTablet state changed. Reason: New leader cde272c487324aa1ac82c3e2bc74267f. Latest consensus state: current_term: 1 leader_uuid: "cde272c487324aa1ac82c3e2bc74267f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } } }
I20260927 10:53:48.259632 22686 tablet_replica.cc:442] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:53:48.259935 22688 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 10:53:48.260151 22688 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 10:53:48.260865 22132 catalog_manager.cc:5495] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f reported cstate change: term changed from 0 to 1, leader changed from <none> to cde272c487324aa1ac82c3e2bc74267f (127.14.68.3). New cstate: current_term: 1 leader_uuid: "cde272c487324aa1ac82c3e2bc74267f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } health_report { overall_health: HEALTHY } } }
I20260927 10:53:48.274855 22641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:48.270522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48638 (local address 127.14.68.3:43703)
0927 10:53:48.270640 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:48.270648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:48.270788 (+ 140us) server_negotiation.cc:407] Connection header received
0927 10:53:48.270940 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:48.270943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:48.270995 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:48.271135 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:48.271394 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.272052 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.273369 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.273627 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.273715 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.274069 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:48.274485 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:48.274594 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:48.274721 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 10:53:48.274769 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 10:53:48.274893 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:48.270480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:43703 (local address 127.0.0.1:48638)
0927 10:53:48.270640 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:48.270657 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:48.270787 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:48.271156 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:48.271159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:48.271176 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:53:48.271285 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:48.271291 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.272216 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.272219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:48.273217 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:48.273229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.273738 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.273742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:48.273832 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.273883 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:53:48.274581 (+ 698us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:53:48.274608 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:53:48.274683 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 10:53:48.274763 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 10:53:48.277436 22431 raft_consensus.cc:1261] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 77162430d03d4d87bbd367f0703a6042 [term 1 FOLLOWER]: Refusing update from remote peer cde272c487324aa1ac82c3e2bc74267f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:48.277684 22291 raft_consensus.cc:1261] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 5729ed1c02e7442fb08b248c96e9f55a [term 1 FOLLOWER]: Refusing update from remote peer cde272c487324aa1ac82c3e2bc74267f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:48.277885 22686 consensus_queue.cc:1035] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f [LEADER]: Connected to new peer: Peer: permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:48.278151 22651 consensus_queue.cc:1035] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:48.279767 22664 tablet_replica.cc:439] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 5729ed1c02e7442fb08b248c96e9f55a: TxnStatusTablet state changed. Reason: New leader cde272c487324aa1ac82c3e2bc74267f. Latest consensus state: current_term: 1 leader_uuid: "cde272c487324aa1ac82c3e2bc74267f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } } }
I20260927 10:53:48.279879 22664 tablet_replica.cc:442] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 5729ed1c02e7442fb08b248c96e9f55a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:53:48.279654 22650 tablet_replica.cc:439] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 77162430d03d4d87bbd367f0703a6042: TxnStatusTablet state changed. Reason: New leader cde272c487324aa1ac82c3e2bc74267f. Latest consensus state: current_term: 1 leader_uuid: "cde272c487324aa1ac82c3e2bc74267f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } } }
I20260927 10:53:48.280036 22650 tablet_replica.cc:442] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 77162430d03d4d87bbd367f0703a6042: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:53:48.281313 22686 tablet_replica.cc:439] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cde272c487324aa1ac82c3e2bc74267f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } } }
I20260927 10:53:48.281411 22686 tablet_replica.cc:442] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:53:48.281736 22686 tablet_replica.cc:439] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cde272c487324aa1ac82c3e2bc74267f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } } }
I20260927 10:53:48.281828 22686 tablet_replica.cc:442] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P cde272c487324aa1ac82c3e2bc74267f: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:53:48.282333 22659 mvcc.cc:205] Tried to move back new op lower bound from 7333914330219888640 to 7333914330148843520. Current Snapshot: MvccSnapshot[applied={T|T < 7333914330219888640}]
I20260927 10:53:48.282465 22664 tablet_replica.cc:439] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 5729ed1c02e7442fb08b248c96e9f55a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cde272c487324aa1ac82c3e2bc74267f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } } }
I20260927 10:53:48.282542 22664 tablet_replica.cc:442] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 5729ed1c02e7442fb08b248c96e9f55a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:53:48.283592 22660 mvcc.cc:205] Tried to move back new op lower bound from 7333914330219888640 to 7333914330148843520. Current Snapshot: MvccSnapshot[applied={T|T < 7333914330219888640}]
I20260927 10:53:48.283788 22650 tablet_replica.cc:439] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 77162430d03d4d87bbd367f0703a6042: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cde272c487324aa1ac82c3e2bc74267f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } } }
I20260927 10:53:48.283879 22650 tablet_replica.cc:442] T 7bd8d9e5ac044672a8a1d6848b8f6ee8 P 77162430d03d4d87bbd367f0703a6042: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:53:48.297976 22641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:48.294447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:36545 (local address 127.14.68.3:43703)
0927 10:53:48.294559 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:48.294563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:48.294792 (+ 229us) server_negotiation.cc:407] Connection header received
0927 10:53:48.294911 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:48.294914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:48.294963 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:48.295022 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:48.295293 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.295847 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.297055 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.297269 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.297339 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.297628 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:48.297687 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:48.297735 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:48.297850 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 10:53:48.297888 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 10:53:48.298260 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:48.294381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:43703 (local address 127.14.68.1:36545)
0927 10:53:48.294710 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:48.294724 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:48.294827 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:48.295065 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:48.295068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:48.295076 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:53:48.295206 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:48.295215 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.295950 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.295954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:48.296938 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:48.296948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.297372 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.297376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:48.297461 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.297512 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:53:48.297766 (+ 254us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:53:48.297772 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:53:48.297835 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 10:53:48.297918 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":92,"threads_started":1}
I20260927 10:53:48.303939 22571 raft_consensus.cc:1261] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P cde272c487324aa1ac82c3e2bc74267f [term 1 FOLLOWER]: Refusing update from remote peer 5729ed1c02e7442fb08b248c96e9f55a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:48.303939 22431 raft_consensus.cc:1261] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 77162430d03d4d87bbd367f0703a6042 [term 1 FOLLOWER]: Refusing update from remote peer 5729ed1c02e7442fb08b248c96e9f55a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:48.304294 22653 consensus_queue.cc:1035] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:48.304422 22664 consensus_queue.cc:1035] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a [LEADER]: Connected to new peer: Peer: permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:48.324695 22642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:48.319617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:56985 (local address 127.14.68.1:35727)
0927 10:53:48.319740 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:48.319744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:48.320531 (+ 787us) server_negotiation.cc:407] Connection header received
0927 10:53:48.320772 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:48.320775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:48.320831 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:48.320908 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:48.321279 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.322074 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.323388 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.323755 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.323823 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.324204 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:48.324273 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:48.324332 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:48.324479 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 10:53:48.324536 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 10:53:48.324695 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:48.319403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:35727 (local address 127.14.68.3:56985)
0927 10:53:48.320455 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:48.320472 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:48.320615 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:48.321015 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:48.321018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:48.321026 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:53:48.321151 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:48.321157 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.322236 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.322239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:48.323255 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:48.323265 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.323924 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.323927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:48.324015 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.324062 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:53:48.324366 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:53:48.324372 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:53:48.324443 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 10:53:48.324521 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"thread_start_us":117,"threads_started":1}
I20260927 10:53:48.342794 22132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:48.342967 22132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:48.344950 22132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc: Sending DeleteTablet for 3 replicas of tablet 2afdc0a5c7c24b08ba972a8e2c3ea31b
I20260927 10:53:48.345507 22271 tablet_service.cc:1514] Processing DeleteTablet for tablet 2afdc0a5c7c24b08ba972a8e2c3ea31b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:48 UTC) from {username='oryx', principal='oryx/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:58002
I20260927 10:53:48.345692 22671 tablet_replica.cc:330] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a: stopping tablet replica
I20260927 10:53:48.345729 22551 tablet_service.cc:1514] Processing DeleteTablet for tablet 2afdc0a5c7c24b08ba972a8e2c3ea31b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:48 UTC) from {username='oryx', principal='oryx/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:48630
I20260927 10:53:48.345773 22411 tablet_service.cc:1514] Processing DeleteTablet for tablet 2afdc0a5c7c24b08ba972a8e2c3ea31b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:48 UTC) from {username='oryx', principal='oryx/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:53596
I20260927 10:53:48.345808 22671 raft_consensus.cc:2229] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:53:48.345943 22670 tablet_replica.cc:330] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 77162430d03d4d87bbd367f0703a6042: stopping tablet replica
I20260927 10:53:48.345955 22669 tablet_replica.cc:330] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P cde272c487324aa1ac82c3e2bc74267f: stopping tablet replica
I20260927 10:53:48.345992 22671 raft_consensus.cc:2258] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:48.346050 22670 raft_consensus.cc:2229] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 77162430d03d4d87bbd367f0703a6042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:48.346083 22669 raft_consensus.cc:2229] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P cde272c487324aa1ac82c3e2bc74267f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:48.346175 22670 raft_consensus.cc:2258] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 77162430d03d4d87bbd367f0703a6042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:48.346191 22669 raft_consensus.cc:2258] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P cde272c487324aa1ac82c3e2bc74267f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:48.346743 22671 ts_tablet_manager.cc:1905] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:48.346951 22670 ts_tablet_manager.cc:1905] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 77162430d03d4d87bbd367f0703a6042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:48.349745 22670 ts_tablet_manager.cc:1918] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 77162430d03d4d87bbd367f0703a6042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:53:48.349819 22670 log.cc:1196] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 77162430d03d4d87bbd367f0703a6042: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/2afdc0a5c7c24b08ba972a8e2c3ea31b
I20260927 10:53:48.350067 22670 ts_tablet_manager.cc:1939] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 77162430d03d4d87bbd367f0703a6042: Deleting consensus metadata
I20260927 10:53:48.350204 22671 ts_tablet_manager.cc:1918] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:53:48.350281 22671 log.cc:1196] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-0/wal/wals/2afdc0a5c7c24b08ba972a8e2c3ea31b
I20260927 10:53:48.350543 22671 ts_tablet_manager.cc:1939] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P 5729ed1c02e7442fb08b248c96e9f55a: Deleting consensus metadata
I20260927 10:53:48.350907 22119 catalog_manager.cc:4841] TS 77162430d03d4d87bbd367f0703a6042 (127.14.68.2:33069): tablet 2afdc0a5c7c24b08ba972a8e2c3ea31b (table test-table [id=30d9c3f5ecad4aec91e0f2c8bbe20a06]) successfully deleted
I20260927 10:53:48.351212 22119 catalog_manager.cc:4841] TS 5729ed1c02e7442fb08b248c96e9f55a (127.14.68.1:35727): tablet 2afdc0a5c7c24b08ba972a8e2c3ea31b (table test-table [id=30d9c3f5ecad4aec91e0f2c8bbe20a06]) successfully deleted
I20260927 10:53:48.351670 22669 ts_tablet_manager.cc:1905] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P cde272c487324aa1ac82c3e2bc74267f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:48.354007 22669 ts_tablet_manager.cc:1918] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P cde272c487324aa1ac82c3e2bc74267f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:53:48.354058 22669 log.cc:1196] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P cde272c487324aa1ac82c3e2bc74267f: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/2afdc0a5c7c24b08ba972a8e2c3ea31b
I20260927 10:53:48.354277 22669 ts_tablet_manager.cc:1939] T 2afdc0a5c7c24b08ba972a8e2c3ea31b P cde272c487324aa1ac82c3e2bc74267f: Deleting consensus metadata
I20260927 10:53:48.354978 22121 catalog_manager.cc:4841] TS cde272c487324aa1ac82c3e2bc74267f (127.14.68.3:43703): tablet 2afdc0a5c7c24b08ba972a8e2c3ea31b (table test-table [id=30d9c3f5ecad4aec91e0f2c8bbe20a06]) successfully deleted
Sep 27 10:53:48 dist-test-slave-tz51 krb5kdc[22089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:53:48 dist-test-slave-tz51 krb5kdc[22089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:48 dist-test-slave-tz51 krb5kdc[22089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 10:53:48.377699 22183 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:48.371257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53746 (local address 127.14.68.62:40641)
0927 10:53:48.371385 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:48.371389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:48.371647 (+ 258us) server_negotiation.cc:407] Connection header received
0927 10:53:48.371849 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:48.371853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:48.371906 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:48.371992 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:48.372702 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.373228 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.374084 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.374312 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.374423 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.377548 (+ 3125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53746: BlockingRecv error: recv got EOF from 127.0.0.1:53746 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 10:53:48 dist-test-slave-tz51 krb5kdc[22089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM
I20260927 10:53:48.387377 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:48.379220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53762 (local address 127.14.68.62:40641)
0927 10:53:48.379349 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:48.379355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:48.379548 (+ 193us) server_negotiation.cc:407] Connection header received
0927 10:53:48.379782 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:48.379785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:48.379827 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:48.379909 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:48.380481 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.380972 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.381759 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.381963 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.382062 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.384262 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:48.384265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:48.384268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:48.384295 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:48.385818 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:48.386247 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:48.386251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:48.386253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:48.386342 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:48.386582 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:48.386585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:48.386587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:48.386801 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:48.386932 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:48.387103 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 10:53:48.387222 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:53:48.389945 22132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:53:48.390404 22132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:48.395299 22271 tablet_service.cc:1467] Processing CreateTablet for tablet 729cfd5bc4454bdea1fccb30f3457b45 (DEFAULT_TABLE table=test-table [id=cfbd461f18a54fbb8e09a5f57826379f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:48.395373 22411 tablet_service.cc:1467] Processing CreateTablet for tablet 729cfd5bc4454bdea1fccb30f3457b45 (DEFAULT_TABLE table=test-table [id=cfbd461f18a54fbb8e09a5f57826379f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:48.395612 22271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 729cfd5bc4454bdea1fccb30f3457b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:48.395684 22411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 729cfd5bc4454bdea1fccb30f3457b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:48.397549 22649 tablet_bootstrap.cc:492] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a: Bootstrap starting.
I20260927 10:53:48.397923 22551 tablet_service.cc:1467] Processing CreateTablet for tablet 729cfd5bc4454bdea1fccb30f3457b45 (DEFAULT_TABLE table=test-table [id=cfbd461f18a54fbb8e09a5f57826379f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:48.398172 22551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 729cfd5bc4454bdea1fccb30f3457b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:48.398180 22645 tablet_bootstrap.cc:492] T 729cfd5bc4454bdea1fccb30f3457b45 P 77162430d03d4d87bbd367f0703a6042: Bootstrap starting.
I20260927 10:53:48.398386 22649 tablet_bootstrap.cc:654] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:48.399068 22645 tablet_bootstrap.cc:654] T 729cfd5bc4454bdea1fccb30f3457b45 P 77162430d03d4d87bbd367f0703a6042: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:48.399550 22649 tablet_bootstrap.cc:492] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a: No bootstrap required, opened a new log
I20260927 10:53:48.399644 22649 ts_tablet_manager.cc:1397] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 10:53:48.400043 22649 raft_consensus.cc:348] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.400158 22649 raft_consensus.cc:374] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:48.400197 22649 raft_consensus.cc:729] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5729ed1c02e7442fb08b248c96e9f55a, State: Initialized, Role: FOLLOWER
I20260927 10:53:48.400303 22649 consensus_queue.cc:260] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.400571 22649 ts_tablet_manager.cc:1428] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:53:48.400661 22646 tablet_bootstrap.cc:492] T 729cfd5bc4454bdea1fccb30f3457b45 P cde272c487324aa1ac82c3e2bc74267f: Bootstrap starting.
I20260927 10:53:48.400668 22645 tablet_bootstrap.cc:492] T 729cfd5bc4454bdea1fccb30f3457b45 P 77162430d03d4d87bbd367f0703a6042: No bootstrap required, opened a new log
I20260927 10:53:48.400754 22645 ts_tablet_manager.cc:1397] T 729cfd5bc4454bdea1fccb30f3457b45 P 77162430d03d4d87bbd367f0703a6042: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 10:53:48.401180 22645 raft_consensus.cc:348] T 729cfd5bc4454bdea1fccb30f3457b45 P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.401290 22645 raft_consensus.cc:374] T 729cfd5bc4454bdea1fccb30f3457b45 P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:48.401320 22645 raft_consensus.cc:729] T 729cfd5bc4454bdea1fccb30f3457b45 P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77162430d03d4d87bbd367f0703a6042, State: Initialized, Role: FOLLOWER
I20260927 10:53:48.401470 22646 tablet_bootstrap.cc:654] T 729cfd5bc4454bdea1fccb30f3457b45 P cde272c487324aa1ac82c3e2bc74267f: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:48.401432 22645 consensus_queue.cc:260] T 729cfd5bc4454bdea1fccb30f3457b45 P 77162430d03d4d87bbd367f0703a6042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.401677 22645 ts_tablet_manager.cc:1428] T 729cfd5bc4454bdea1fccb30f3457b45 P 77162430d03d4d87bbd367f0703a6042: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:53:48.402637 22646 tablet_bootstrap.cc:492] T 729cfd5bc4454bdea1fccb30f3457b45 P cde272c487324aa1ac82c3e2bc74267f: No bootstrap required, opened a new log
I20260927 10:53:48.402727 22646 ts_tablet_manager.cc:1397] T 729cfd5bc4454bdea1fccb30f3457b45 P cde272c487324aa1ac82c3e2bc74267f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:53:48.403132 22646 raft_consensus.cc:348] T 729cfd5bc4454bdea1fccb30f3457b45 P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.403257 22646 raft_consensus.cc:374] T 729cfd5bc4454bdea1fccb30f3457b45 P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:48.403290 22646 raft_consensus.cc:729] T 729cfd5bc4454bdea1fccb30f3457b45 P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde272c487324aa1ac82c3e2bc74267f, State: Initialized, Role: FOLLOWER
I20260927 10:53:48.403403 22646 consensus_queue.cc:260] T 729cfd5bc4454bdea1fccb30f3457b45 P cde272c487324aa1ac82c3e2bc74267f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.403656 22646 ts_tablet_manager.cc:1428] T 729cfd5bc4454bdea1fccb30f3457b45 P cde272c487324aa1ac82c3e2bc74267f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:53:48.528955 22656 raft_consensus.cc:482] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:48.529146 22656 raft_consensus.cc:504] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.529601 22656 leader_election.cc:288] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77162430d03d4d87bbd367f0703a6042 (127.14.68.2:33069), cde272c487324aa1ac82c3e2bc74267f (127.14.68.3:43703)
I20260927 10:53:48.530015 22431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "729cfd5bc4454bdea1fccb30f3457b45" candidate_uuid: "5729ed1c02e7442fb08b248c96e9f55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77162430d03d4d87bbd367f0703a6042" is_pre_election: true
I20260927 10:53:48.530086 22571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "729cfd5bc4454bdea1fccb30f3457b45" candidate_uuid: "5729ed1c02e7442fb08b248c96e9f55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde272c487324aa1ac82c3e2bc74267f" is_pre_election: true
I20260927 10:53:48.530192 22431 raft_consensus.cc:2454] T 729cfd5bc4454bdea1fccb30f3457b45 P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5729ed1c02e7442fb08b248c96e9f55a in term 0.
I20260927 10:53:48.530260 22571 raft_consensus.cc:2454] T 729cfd5bc4454bdea1fccb30f3457b45 P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5729ed1c02e7442fb08b248c96e9f55a in term 0.
I20260927 10:53:48.530503 22223 leader_election.cc:302] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5729ed1c02e7442fb08b248c96e9f55a, 77162430d03d4d87bbd367f0703a6042; no voters:
I20260927 10:53:48.530717 22656 raft_consensus.cc:2781] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:48.530781 22656 raft_consensus.cc:482] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:48.530836 22656 raft_consensus.cc:3037] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:48.531935 22656 raft_consensus.cc:504] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.532264 22656 leader_election.cc:288] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a [CANDIDATE]: Term 1 election: Requested vote from peers 77162430d03d4d87bbd367f0703a6042 (127.14.68.2:33069), cde272c487324aa1ac82c3e2bc74267f (127.14.68.3:43703)
I20260927 10:53:48.532660 22431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "729cfd5bc4454bdea1fccb30f3457b45" candidate_uuid: "5729ed1c02e7442fb08b248c96e9f55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77162430d03d4d87bbd367f0703a6042"
I20260927 10:53:48.532686 22571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "729cfd5bc4454bdea1fccb30f3457b45" candidate_uuid: "5729ed1c02e7442fb08b248c96e9f55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde272c487324aa1ac82c3e2bc74267f"
I20260927 10:53:48.532766 22431 raft_consensus.cc:3037] T 729cfd5bc4454bdea1fccb30f3457b45 P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:48.532814 22571 raft_consensus.cc:3037] T 729cfd5bc4454bdea1fccb30f3457b45 P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:48.533700 22571 raft_consensus.cc:2454] T 729cfd5bc4454bdea1fccb30f3457b45 P cde272c487324aa1ac82c3e2bc74267f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5729ed1c02e7442fb08b248c96e9f55a in term 1.
I20260927 10:53:48.533715 22431 raft_consensus.cc:2454] T 729cfd5bc4454bdea1fccb30f3457b45 P 77162430d03d4d87bbd367f0703a6042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5729ed1c02e7442fb08b248c96e9f55a in term 1.
I20260927 10:53:48.534001 22225 leader_election.cc:302] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5729ed1c02e7442fb08b248c96e9f55a, cde272c487324aa1ac82c3e2bc74267f; no voters:
I20260927 10:53:48.534168 22656 raft_consensus.cc:2781] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:48.534253 22656 raft_consensus.cc:686] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a [term 1 LEADER]: Becoming Leader. State: Replica: 5729ed1c02e7442fb08b248c96e9f55a, State: Running, Role: LEADER
I20260927 10:53:48.534407 22656 consensus_queue.cc:237] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.536126 22133 catalog_manager.cc:5495] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5729ed1c02e7442fb08b248c96e9f55a (127.14.68.1). New cstate: current_term: 1 leader_uuid: "5729ed1c02e7442fb08b248c96e9f55a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } health_report { overall_health: UNKNOWN } } }
I20260927 10:53:48.560467 22642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:48.557149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58016 (local address 127.14.68.1:35727)
0927 10:53:48.557264 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:48.557268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:48.557279 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:53:48.557432 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:48.557435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:48.557508 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:48.557588 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:48.557811 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.558292 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.559405 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.559621 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.559716 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.560006 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:48.560068 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:48.560117 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:48.560256 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 10:53:48.560298 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:53:48.562346 22571 raft_consensus.cc:1261] T 729cfd5bc4454bdea1fccb30f3457b45 P cde272c487324aa1ac82c3e2bc74267f [term 1 FOLLOWER]: Refusing update from remote peer 5729ed1c02e7442fb08b248c96e9f55a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:48.562446 22431 raft_consensus.cc:1261] T 729cfd5bc4454bdea1fccb30f3457b45 P 77162430d03d4d87bbd367f0703a6042 [term 1 FOLLOWER]: Refusing update from remote peer 5729ed1c02e7442fb08b248c96e9f55a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:48.562850 22664 consensus_queue.cc:1035] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:48.562992 22656 consensus_queue.cc:1035] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a [LEADER]: Connected to new peer: Peer: permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:48.570035 22133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:48.570242 22133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:48.572103 22133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc: Sending DeleteTablet for 3 replicas of tablet 729cfd5bc4454bdea1fccb30f3457b45
I20260927 10:53:48.572566 22271 tablet_service.cc:1514] Processing DeleteTablet for tablet 729cfd5bc4454bdea1fccb30f3457b45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:48 UTC) from {username='oryx', principal='oryx/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:58002
I20260927 10:53:48.572762 22671 tablet_replica.cc:330] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a: stopping tablet replica
I20260927 10:53:48.572888 22671 raft_consensus.cc:2229] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:53:48.573067 22671 raft_consensus.cc:2258] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:48.573218 22411 tablet_service.cc:1514] Processing DeleteTablet for tablet 729cfd5bc4454bdea1fccb30f3457b45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:48 UTC) from {username='oryx', principal='oryx/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:53596
I20260927 10:53:48.573468 22670 tablet_replica.cc:330] T 729cfd5bc4454bdea1fccb30f3457b45 P 77162430d03d4d87bbd367f0703a6042: stopping tablet replica
I20260927 10:53:48.573577 22670 raft_consensus.cc:2229] T 729cfd5bc4454bdea1fccb30f3457b45 P 77162430d03d4d87bbd367f0703a6042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:48.573704 22670 raft_consensus.cc:2258] T 729cfd5bc4454bdea1fccb30f3457b45 P 77162430d03d4d87bbd367f0703a6042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:48.574827 22670 ts_tablet_manager.cc:1905] T 729cfd5bc4454bdea1fccb30f3457b45 P 77162430d03d4d87bbd367f0703a6042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:48.575911 22551 tablet_service.cc:1514] Processing DeleteTablet for tablet 729cfd5bc4454bdea1fccb30f3457b45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:48 UTC) from {username='oryx', principal='oryx/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:48630
I20260927 10:53:48.576064 22669 tablet_replica.cc:330] T 729cfd5bc4454bdea1fccb30f3457b45 P cde272c487324aa1ac82c3e2bc74267f: stopping tablet replica
I20260927 10:53:48.576110 22671 ts_tablet_manager.cc:1905] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:48.576182 22669 raft_consensus.cc:2229] T 729cfd5bc4454bdea1fccb30f3457b45 P cde272c487324aa1ac82c3e2bc74267f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:48.576301 22669 raft_consensus.cc:2258] T 729cfd5bc4454bdea1fccb30f3457b45 P cde272c487324aa1ac82c3e2bc74267f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:48.577018 22133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 10:53:48.577112 22669 ts_tablet_manager.cc:1905] T 729cfd5bc4454bdea1fccb30f3457b45 P cde272c487324aa1ac82c3e2bc74267f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:48.577313 22670 ts_tablet_manager.cc:1918] T 729cfd5bc4454bdea1fccb30f3457b45 P 77162430d03d4d87bbd367f0703a6042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:48.577390 22670 log.cc:1196] T 729cfd5bc4454bdea1fccb30f3457b45 P 77162430d03d4d87bbd367f0703a6042: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/729cfd5bc4454bdea1fccb30f3457b45
I20260927 10:53:48.577631 22670 ts_tablet_manager.cc:1939] T 729cfd5bc4454bdea1fccb30f3457b45 P 77162430d03d4d87bbd367f0703a6042: Deleting consensus metadata
W20260927 10:53:48.577849 22133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:53:48.578395 22671 ts_tablet_manager.cc:1918] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:48.578471 22671 log.cc:1196] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-0/wal/wals/729cfd5bc4454bdea1fccb30f3457b45
I20260927 10:53:48.578720 22671 ts_tablet_manager.cc:1939] T 729cfd5bc4454bdea1fccb30f3457b45 P 5729ed1c02e7442fb08b248c96e9f55a: Deleting consensus metadata
I20260927 10:53:48.578814 22119 catalog_manager.cc:4841] TS 77162430d03d4d87bbd367f0703a6042 (127.14.68.2:33069): tablet 729cfd5bc4454bdea1fccb30f3457b45 (table test-table [id=cfbd461f18a54fbb8e09a5f57826379f]) successfully deleted
I20260927 10:53:48.579300 22119 catalog_manager.cc:4841] TS 5729ed1c02e7442fb08b248c96e9f55a (127.14.68.1:35727): tablet 729cfd5bc4454bdea1fccb30f3457b45 (table test-table [id=cfbd461f18a54fbb8e09a5f57826379f]) successfully deleted
I20260927 10:53:48.579550 22669 ts_tablet_manager.cc:1918] T 729cfd5bc4454bdea1fccb30f3457b45 P cde272c487324aa1ac82c3e2bc74267f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:53:48.579618 22669 log.cc:1196] T 729cfd5bc4454bdea1fccb30f3457b45 P cde272c487324aa1ac82c3e2bc74267f: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/729cfd5bc4454bdea1fccb30f3457b45
I20260927 10:53:48.579892 22669 ts_tablet_manager.cc:1939] T 729cfd5bc4454bdea1fccb30f3457b45 P cde272c487324aa1ac82c3e2bc74267f: Deleting consensus metadata
I20260927 10:53:48.580546 22121 catalog_manager.cc:4841] TS cde272c487324aa1ac82c3e2bc74267f (127.14.68.3:43703): tablet 729cfd5bc4454bdea1fccb30f3457b45 (table test-table [id=cfbd461f18a54fbb8e09a5f57826379f]) successfully deleted
I20260927 10:53:48.582284 22271 tablet_service.cc:1467] Processing CreateTablet for tablet 30b6edd7a44d4b739dcbac93cbdeb73e (DEFAULT_TABLE table=test-table [id=9efa661e71bb42598115fac86ca887c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:48.582304 22411 tablet_service.cc:1467] Processing CreateTablet for tablet 30b6edd7a44d4b739dcbac93cbdeb73e (DEFAULT_TABLE table=test-table [id=9efa661e71bb42598115fac86ca887c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:48.582435 22551 tablet_service.cc:1467] Processing CreateTablet for tablet 30b6edd7a44d4b739dcbac93cbdeb73e (DEFAULT_TABLE table=test-table [id=9efa661e71bb42598115fac86ca887c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:53:48.582512 22271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30b6edd7a44d4b739dcbac93cbdeb73e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:48.582522 22411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30b6edd7a44d4b739dcbac93cbdeb73e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:48.582646 22551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30b6edd7a44d4b739dcbac93cbdeb73e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:48.584599 22649 tablet_bootstrap.cc:492] T 30b6edd7a44d4b739dcbac93cbdeb73e P 5729ed1c02e7442fb08b248c96e9f55a: Bootstrap starting.
I20260927 10:53:48.584594 22646 tablet_bootstrap.cc:492] T 30b6edd7a44d4b739dcbac93cbdeb73e P cde272c487324aa1ac82c3e2bc74267f: Bootstrap starting.
I20260927 10:53:48.584621 22645 tablet_bootstrap.cc:492] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042: Bootstrap starting.
I20260927 10:53:48.585304 22645 tablet_bootstrap.cc:654] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:48.585359 22649 tablet_bootstrap.cc:654] T 30b6edd7a44d4b739dcbac93cbdeb73e P 5729ed1c02e7442fb08b248c96e9f55a: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:48.585359 22646 tablet_bootstrap.cc:654] T 30b6edd7a44d4b739dcbac93cbdeb73e P cde272c487324aa1ac82c3e2bc74267f: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:48.586390 22645 tablet_bootstrap.cc:492] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042: No bootstrap required, opened a new log
I20260927 10:53:48.586473 22645 ts_tablet_manager.cc:1397] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:53:48.586704 22646 tablet_bootstrap.cc:492] T 30b6edd7a44d4b739dcbac93cbdeb73e P cde272c487324aa1ac82c3e2bc74267f: No bootstrap required, opened a new log
I20260927 10:53:48.586779 22646 ts_tablet_manager.cc:1397] T 30b6edd7a44d4b739dcbac93cbdeb73e P cde272c487324aa1ac82c3e2bc74267f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 10:53:48.586903 22645 raft_consensus.cc:348] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.587009 22645 raft_consensus.cc:374] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:48.587023 22649 tablet_bootstrap.cc:492] T 30b6edd7a44d4b739dcbac93cbdeb73e P 5729ed1c02e7442fb08b248c96e9f55a: No bootstrap required, opened a new log
I20260927 10:53:48.587047 22645 raft_consensus.cc:729] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77162430d03d4d87bbd367f0703a6042, State: Initialized, Role: FOLLOWER
I20260927 10:53:48.587100 22649 ts_tablet_manager.cc:1397] T 30b6edd7a44d4b739dcbac93cbdeb73e P 5729ed1c02e7442fb08b248c96e9f55a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 10:53:48.587167 22645 consensus_queue.cc:260] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.587167 22646 raft_consensus.cc:348] T 30b6edd7a44d4b739dcbac93cbdeb73e P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.587261 22646 raft_consensus.cc:374] T 30b6edd7a44d4b739dcbac93cbdeb73e P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:48.587302 22646 raft_consensus.cc:729] T 30b6edd7a44d4b739dcbac93cbdeb73e P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde272c487324aa1ac82c3e2bc74267f, State: Initialized, Role: FOLLOWER
I20260927 10:53:48.587414 22646 consensus_queue.cc:260] T 30b6edd7a44d4b739dcbac93cbdeb73e P cde272c487324aa1ac82c3e2bc74267f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.587469 22645 ts_tablet_manager.cc:1428] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:53:48.587476 22649 raft_consensus.cc:348] T 30b6edd7a44d4b739dcbac93cbdeb73e P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.587606 22649 raft_consensus.cc:374] T 30b6edd7a44d4b739dcbac93cbdeb73e P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:48.587649 22649 raft_consensus.cc:729] T 30b6edd7a44d4b739dcbac93cbdeb73e P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5729ed1c02e7442fb08b248c96e9f55a, State: Initialized, Role: FOLLOWER
I20260927 10:53:48.587677 22646 ts_tablet_manager.cc:1428] T 30b6edd7a44d4b739dcbac93cbdeb73e P cde272c487324aa1ac82c3e2bc74267f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 10:53:48.587800 22649 consensus_queue.cc:260] T 30b6edd7a44d4b739dcbac93cbdeb73e P 5729ed1c02e7442fb08b248c96e9f55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.588061 22649 ts_tablet_manager.cc:1428] T 30b6edd7a44d4b739dcbac93cbdeb73e P 5729ed1c02e7442fb08b248c96e9f55a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 10:53:48.618505 22650 raft_consensus.cc:482] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:53:48.618640 22650 raft_consensus.cc:504] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.619760 22650 leader_election.cc:288] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5729ed1c02e7442fb08b248c96e9f55a (127.14.68.1:35727), cde272c487324aa1ac82c3e2bc74267f (127.14.68.3:43703)
I20260927 10:53:48.624190 22641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:48.620072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:35901 (local address 127.14.68.3:43703)
0927 10:53:48.620195 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:48.620200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:48.620459 (+ 259us) server_negotiation.cc:407] Connection header received
0927 10:53:48.620563 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:48.620565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:48.620636 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:48.620712 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:48.620988 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.621646 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.622997 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.623586 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.623675 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.623803 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:48.623985 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 10:53:48.624072 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:53:48.624217 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:48.619969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:43703 (local address 127.14.68.2:35901)
0927 10:53:48.620344 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:48.620359 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:48.620467 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:48.620745 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:48.620747 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:48.620761 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:48.620876 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:48.620883 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.621743 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.621746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:48.622864 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:48.622873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.623751 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.623759 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:48.623851 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.623879 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 10:53:48.623964 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 10:53:48.624033 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":74,"thread_start_us":73,"threads_started":1}
I20260927 10:53:48.624233 22642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:48.620058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:42275 (local address 127.14.68.1:35727)
0927 10:53:48.620185 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:48.620192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:48.620357 (+ 165us) server_negotiation.cc:407] Connection header received
0927 10:53:48.620475 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:48.620479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:48.620535 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:48.620611 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:53:48.620914 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.621578 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.623108 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.623622 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.623799 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.623861 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:48.624040 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 10:53:48.624115 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:53:48.624253 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:48.619957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:35727 (local address 127.14.68.2:42275)
0927 10:53:48.620270 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:48.620282 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:48.620370 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:48.620634 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:48.620637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:48.620655 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:53:48.620797 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:48.620806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.621706 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.621711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:48.622913 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:48.622940 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.623801 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.623804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:48.623919 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.623947 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 10:53:48.624022 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 10:53:48.624091 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":54,"thread_start_us":101,"threads_started":1}
I20260927 10:53:48.624734 22571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30b6edd7a44d4b739dcbac93cbdeb73e" candidate_uuid: "77162430d03d4d87bbd367f0703a6042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde272c487324aa1ac82c3e2bc74267f" is_pre_election: true
I20260927 10:53:48.624856 22291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30b6edd7a44d4b739dcbac93cbdeb73e" candidate_uuid: "77162430d03d4d87bbd367f0703a6042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5729ed1c02e7442fb08b248c96e9f55a" is_pre_election: true
I20260927 10:53:48.624908 22571 raft_consensus.cc:2454] T 30b6edd7a44d4b739dcbac93cbdeb73e P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77162430d03d4d87bbd367f0703a6042 in term 0.
I20260927 10:53:48.625029 22291 raft_consensus.cc:2454] T 30b6edd7a44d4b739dcbac93cbdeb73e P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77162430d03d4d87bbd367f0703a6042 in term 0.
I20260927 10:53:48.625334 22365 leader_election.cc:302] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5729ed1c02e7442fb08b248c96e9f55a, 77162430d03d4d87bbd367f0703a6042, cde272c487324aa1ac82c3e2bc74267f; no voters:
I20260927 10:53:48.625644 22650 raft_consensus.cc:2781] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:53:48.625691 22650 raft_consensus.cc:482] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:53:48.625717 22650 raft_consensus.cc:3037] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:48.626714 22650 raft_consensus.cc:504] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.627055 22650 leader_election.cc:288] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042 [CANDIDATE]: Term 1 election: Requested vote from peers 5729ed1c02e7442fb08b248c96e9f55a (127.14.68.1:35727), cde272c487324aa1ac82c3e2bc74267f (127.14.68.3:43703)
I20260927 10:53:48.627416 22571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30b6edd7a44d4b739dcbac93cbdeb73e" candidate_uuid: "77162430d03d4d87bbd367f0703a6042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde272c487324aa1ac82c3e2bc74267f"
I20260927 10:53:48.627539 22571 raft_consensus.cc:3037] T 30b6edd7a44d4b739dcbac93cbdeb73e P cde272c487324aa1ac82c3e2bc74267f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:48.628432 22571 raft_consensus.cc:2454] T 30b6edd7a44d4b739dcbac93cbdeb73e P cde272c487324aa1ac82c3e2bc74267f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77162430d03d4d87bbd367f0703a6042 in term 1.
I20260927 10:53:48.628707 22291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30b6edd7a44d4b739dcbac93cbdeb73e" candidate_uuid: "77162430d03d4d87bbd367f0703a6042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5729ed1c02e7442fb08b248c96e9f55a"
I20260927 10:53:48.628738 22365 leader_election.cc:302] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77162430d03d4d87bbd367f0703a6042, cde272c487324aa1ac82c3e2bc74267f; no voters:
I20260927 10:53:48.628836 22291 raft_consensus.cc:3037] T 30b6edd7a44d4b739dcbac93cbdeb73e P 5729ed1c02e7442fb08b248c96e9f55a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:53:48.629684 22291 raft_consensus.cc:2454] T 30b6edd7a44d4b739dcbac93cbdeb73e P 5729ed1c02e7442fb08b248c96e9f55a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77162430d03d4d87bbd367f0703a6042 in term 1.
I20260927 10:53:48.629877 22650 raft_consensus.cc:2781] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:48.630131 22650 raft_consensus.cc:686] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042 [term 1 LEADER]: Becoming Leader. State: Replica: 77162430d03d4d87bbd367f0703a6042, State: Running, Role: LEADER
I20260927 10:53:48.630391 22650 consensus_queue.cc:237] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } }
I20260927 10:53:48.632736 22134 catalog_manager.cc:5495] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77162430d03d4d87bbd367f0703a6042 (127.14.68.2). New cstate: current_term: 1 leader_uuid: "77162430d03d4d87bbd367f0703a6042" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77162430d03d4d87bbd367f0703a6042" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 33069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 } health_report { overall_health: UNKNOWN } } }
I20260927 10:53:48.657128 22643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:48.653138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53610 (local address 127.14.68.2:33069)
0927 10:53:48.653261 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:48.653265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:48.653351 (+ 86us) server_negotiation.cc:407] Connection header received
0927 10:53:48.653543 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:48.653548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:48.653605 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:48.653727 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:48.653956 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.654517 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.655394 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.655777 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.655887 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.656174 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:48.656733 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:48.656858 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:48.656938 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 10:53:48.656990 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:53:48.666957 22641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:48.663739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:59045 (local address 127.14.68.3:43703)
0927 10:53:48.663879 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:48.663885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:48.664009 (+ 124us) server_negotiation.cc:407] Connection header received
0927 10:53:48.664143 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:48.664146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:48.664196 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:48.664279 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:48.664525 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.665017 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.665882 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.666129 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.666241 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.666532 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:48.666596 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:48.666686 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:48.666772 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 10:53:48.666818 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:53:48.666993 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:48.663567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:43703 (local address 127.14.68.2:59045)
0927 10:53:48.663916 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:48.663930 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:48.664051 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:48.664291 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:48.664294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:48.664301 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:53:48.664415 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:48.664425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.665129 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.665134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:48.665766 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:48.665774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.666247 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.666251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:48.666344 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.666391 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:53:48.666688 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:53:48.666696 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 10:53:48.666762 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 10:53:48.666839 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":155,"threads_started":1}
I20260927 10:53:48.673009 22571 raft_consensus.cc:1261] T 30b6edd7a44d4b739dcbac93cbdeb73e P cde272c487324aa1ac82c3e2bc74267f [term 1 FOLLOWER]: Refusing update from remote peer 77162430d03d4d87bbd367f0703a6042: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:48.673377 22291 raft_consensus.cc:1261] T 30b6edd7a44d4b739dcbac93cbdeb73e P 5729ed1c02e7442fb08b248c96e9f55a [term 1 FOLLOWER]: Refusing update from remote peer 77162430d03d4d87bbd367f0703a6042: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:53:48.673596 22650 consensus_queue.cc:1035] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cde272c487324aa1ac82c3e2bc74267f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:48.673813 22727 consensus_queue.cc:1035] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5729ed1c02e7442fb08b248c96e9f55a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 35727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:53:48.690562 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:48.687507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:33069 (local address 127.14.68.3:38757)
0927 10:53:48.687615 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:48.687631 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:48.687732 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:48.687960 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:48.687963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:48.687971 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:53:48.688070 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:48.688076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.688802 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.688806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:48.689411 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:48.689420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.689884 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.689888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:48.689974 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.690009 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:53:48.690292 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:53:48.690301 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 10:53:48.690362 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 10:53:48.690444 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 10:53:48.690562 22643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:48.687550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:38757 (local address 127.14.68.2:33069)
0927 10:53:48.687649 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:48.687653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:48.687672 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:53:48.687820 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:48.687823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:48.687860 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:48.687948 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:53:48.688184 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.688691 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.689540 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:48.689774 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:48.689884 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:48.690127 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:53:48.690208 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:53:48.690286 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:48.690381 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 10:53:48.690430 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 10:53:48.712972 22134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:48.713110 22134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:53:48.715209 22134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abc739bf6a844b7db4a2d03c71ef39cc: Sending DeleteTablet for 3 replicas of tablet 30b6edd7a44d4b739dcbac93cbdeb73e
I20260927 10:53:48.715826 22411 tablet_service.cc:1514] Processing DeleteTablet for tablet 30b6edd7a44d4b739dcbac93cbdeb73e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:48 UTC) from {username='oryx', principal='oryx/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:53596
I20260927 10:53:48.715870 22271 tablet_service.cc:1514] Processing DeleteTablet for tablet 30b6edd7a44d4b739dcbac93cbdeb73e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:48 UTC) from {username='oryx', principal='oryx/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:58002
I20260927 10:53:48.716033 22671 tablet_replica.cc:330] T 30b6edd7a44d4b739dcbac93cbdeb73e P 5729ed1c02e7442fb08b248c96e9f55a: stopping tablet replica
I20260927 10:53:48.716033 22670 tablet_replica.cc:330] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042: stopping tablet replica
I20260927 10:53:48.716142 22671 raft_consensus.cc:2229] T 30b6edd7a44d4b739dcbac93cbdeb73e P 5729ed1c02e7442fb08b248c96e9f55a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:48.716142 22670 raft_consensus.cc:2229] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:53:48.716257 22671 raft_consensus.cc:2258] T 30b6edd7a44d4b739dcbac93cbdeb73e P 5729ed1c02e7442fb08b248c96e9f55a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:48.716462 22670 raft_consensus.cc:2258] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:48.717085 22671 ts_tablet_manager.cc:1905] T 30b6edd7a44d4b739dcbac93cbdeb73e P 5729ed1c02e7442fb08b248c96e9f55a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:48.717149 22670 ts_tablet_manager.cc:1905] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:48.719101 22671 ts_tablet_manager.cc:1918] T 30b6edd7a44d4b739dcbac93cbdeb73e P 5729ed1c02e7442fb08b248c96e9f55a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:53:48.719205 22671 log.cc:1196] T 30b6edd7a44d4b739dcbac93cbdeb73e P 5729ed1c02e7442fb08b248c96e9f55a: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-0/wal/wals/30b6edd7a44d4b739dcbac93cbdeb73e
I20260927 10:53:48.719437 22671 ts_tablet_manager.cc:1939] T 30b6edd7a44d4b739dcbac93cbdeb73e P 5729ed1c02e7442fb08b248c96e9f55a: Deleting consensus metadata
I20260927 10:53:48.719632 22670 ts_tablet_manager.cc:1918] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:53:48.719736 22670 log.cc:1196] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/30b6edd7a44d4b739dcbac93cbdeb73e
I20260927 10:53:48.720005 22670 ts_tablet_manager.cc:1939] T 30b6edd7a44d4b739dcbac93cbdeb73e P 77162430d03d4d87bbd367f0703a6042: Deleting consensus metadata
I20260927 10:53:48.720182 22119 catalog_manager.cc:4841] TS 5729ed1c02e7442fb08b248c96e9f55a (127.14.68.1:35727): tablet 30b6edd7a44d4b739dcbac93cbdeb73e (table test-table [id=9efa661e71bb42598115fac86ca887c5]) successfully deleted
I20260927 10:53:48.720240 22551 tablet_service.cc:1514] Processing DeleteTablet for tablet 30b6edd7a44d4b739dcbac93cbdeb73e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:53:48 UTC) from {username='oryx', principal='oryx/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:48630
I20260927 10:53:48.720409 22669 tablet_replica.cc:330] T 30b6edd7a44d4b739dcbac93cbdeb73e P cde272c487324aa1ac82c3e2bc74267f: stopping tablet replica
I20260927 10:53:48.720520 22669 raft_consensus.cc:2229] T 30b6edd7a44d4b739dcbac93cbdeb73e P cde272c487324aa1ac82c3e2bc74267f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:53:48.720574 22119 catalog_manager.cc:4841] TS 77162430d03d4d87bbd367f0703a6042 (127.14.68.2:33069): tablet 30b6edd7a44d4b739dcbac93cbdeb73e (table test-table [id=9efa661e71bb42598115fac86ca887c5]) successfully deleted
I20260927 10:53:48.720641 22669 raft_consensus.cc:2258] T 30b6edd7a44d4b739dcbac93cbdeb73e P cde272c487324aa1ac82c3e2bc74267f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:53:48.721395 22669 ts_tablet_manager.cc:1905] T 30b6edd7a44d4b739dcbac93cbdeb73e P cde272c487324aa1ac82c3e2bc74267f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:53:48.723485 22669 ts_tablet_manager.cc:1918] T 30b6edd7a44d4b739dcbac93cbdeb73e P cde272c487324aa1ac82c3e2bc74267f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:53:48.723572 22669 log.cc:1196] T 30b6edd7a44d4b739dcbac93cbdeb73e P cde272c487324aa1ac82c3e2bc74267f: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/30b6edd7a44d4b739dcbac93cbdeb73e
I20260927 10:53:48.723873 22669 ts_tablet_manager.cc:1939] T 30b6edd7a44d4b739dcbac93cbdeb73e P cde272c487324aa1ac82c3e2bc74267f: Deleting consensus metadata
I20260927 10:53:48.724516 22121 catalog_manager.cc:4841] TS cde272c487324aa1ac82c3e2bc74267f (127.14.68.3:43703): tablet 30b6edd7a44d4b739dcbac93cbdeb73e (table test-table [id=9efa661e71bb42598115fac86ca887c5]) successfully deleted
I20260927 10:53:48.725021 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 22206
I20260927 10:53:48.734411 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 22346
I20260927 10:53:48.740691 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 22486
I20260927 10:53:48.746996 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 22102
2026-09-27T10:53:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3015 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:53:48 dist-test-slave-tz51 krb5kdc[22738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:53:48 dist-test-slave-tz51 krb5kdc[22738](info): set up 2 sockets
Sep 27 10:53:48 dist-test-slave-tz51 krb5kdc[22738](info): commencing operation
krb5kdc: starting...
W20260927 10:53:49.780329 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:53:49 dist-test-slave-tz51 krb5kdc[22738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:53:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:53:49.898048 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:37187
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:35363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:37187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:50.009033 22751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:50.009303 22751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:50.009344 22751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:50.013305 22751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:50.013355 22751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:50.013373 22751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:50.013391 22751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:50.013409 22751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:50.018610 22751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:35363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:37187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:37187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:50.019569 22751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:50.020586 22751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:53:50.028967 22751 server_base.cc:1034] running on GCE node
W20260927 10:53:50.028970 22757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:50.029137 22759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:50.028997 22756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:50.029779 22751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:50.030797 22751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:50.031949 22751 hybrid_clock.cc:648] HybridClock initialized: now 1790506430031921 us; error 40 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T10:53:50Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1288 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:53:50 dist-test-slave-tz51 krb5kdc[22764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:53:50 dist-test-slave-tz51 krb5kdc[22764](info): set up 2 sockets
Sep 27 10:53:50 dist-test-slave-tz51 krb5kdc[22764](info): commencing operation
krb5kdc: starting...
W20260927 10:53:51.072440 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:53:51 dist-test-slave-tz51 krb5kdc[22764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:53:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:53:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:53:51.195182 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:40659
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:42709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:40659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:51.302307 22777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:51.302613 22777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:51.302687 22777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:51.306699 22777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:51.306800 22777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:51.306839 22777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:51.306876 22777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:51.306910 22777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:51.312144 22777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:42709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:40659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:40659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:51.313206 22777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:51.314220 22777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:51.322614 22783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:51.322631 22785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:51.322667 22782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:51.322899 22777 server_base.cc:1034] running on GCE node
I20260927 10:53:51.323462 22777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:51.324609 22777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:51.325804 22777 hybrid_clock.cc:648] HybridClock initialized: now 1790506431325789 us; error 28 us; skew 500 ppm
Sep 27 10:53:51 dist-test-slave-tz51 krb5kdc[22764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506431, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:51.332564 22777 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:53:51.333554 22777 webserver.cc:460] Webserver started at http://127.14.68.62:34123/ using document root <none> and password file <none>
I20260927 10:53:51.334136 22777 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:51.334231 22777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:51.334466 22777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:51.336612 22777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "da545bfd50e24c45bf4f8fd92b67f3a4"
format_stamp: "Formatted at 2026-09-27 10:53:51 on dist-test-slave-tz51"
server_key: "68d5ddd00ca2c8aaad31080a167acbb0"
server_key_iv: "8a73a47f5ba5b5a731ec446f4b2dacc9"
server_key_version: "encryptionkey@0"
I20260927 10:53:51.337188 22777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "da545bfd50e24c45bf4f8fd92b67f3a4"
format_stamp: "Formatted at 2026-09-27 10:53:51 on dist-test-slave-tz51"
server_key: "68d5ddd00ca2c8aaad31080a167acbb0"
server_key_iv: "8a73a47f5ba5b5a731ec446f4b2dacc9"
server_key_version: "encryptionkey@0"
I20260927 10:53:51.341154 22777 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:53:51.344199 22792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:51.345563 22777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:53:51.345716 22777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "da545bfd50e24c45bf4f8fd92b67f3a4"
format_stamp: "Formatted at 2026-09-27 10:53:51 on dist-test-slave-tz51"
server_key: "68d5ddd00ca2c8aaad31080a167acbb0"
server_key_iv: "8a73a47f5ba5b5a731ec446f4b2dacc9"
server_key_version: "encryptionkey@0"
I20260927 10:53:51.345849 22777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:51.359552 22777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:51.362874 22777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:51.363088 22777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:51.370034 22777 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:40659
I20260927 10:53:51.370046 22844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:40659 every 8 connection(s)
I20260927 10:53:51.370963 22777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:51.373764 22845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:53:51.379649 22845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4: Bootstrap starting.
I20260927 10:53:51.381253 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 22777
I20260927 10:53:51.381392 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:53:51.381703 14608 external_mini_cluster.cc:1428] Setting key 42fff7fa2688e280871b22203c50e19a
I20260927 10:53:51.382349 22845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4: Neither blocks nor log segments found. Creating new log.
I20260927 10:53:51.383287 22845 log.cc:824] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:51.385285 22845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4: No bootstrap required, opened a new log
I20260927 10:53:51.388679 22845 raft_consensus.cc:348] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da545bfd50e24c45bf4f8fd92b67f3a4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40659 } }
I20260927 10:53:51.388912 22845 raft_consensus.cc:374] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:53:51.388972 22845 raft_consensus.cc:729] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da545bfd50e24c45bf4f8fd92b67f3a4, State: Initialized, Role: FOLLOWER
I20260927 10:53:51.389403 22845 consensus_queue.cc:260] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da545bfd50e24c45bf4f8fd92b67f3a4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40659 } }
I20260927 10:53:51.389518 22845 raft_consensus.cc:388] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:51.389597 22845 raft_consensus.cc:482] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:51.389696 22845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 10:53:51 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:51.390785 22845 raft_consensus.cc:504] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da545bfd50e24c45bf4f8fd92b67f3a4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40659 } }
I20260927 10:53:51.391142 22845 leader_election.cc:302] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da545bfd50e24c45bf4f8fd92b67f3a4; no voters:
I20260927 10:53:51.391502 22845 leader_election.cc:288] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:53:51.391618 22850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:53:51.391872 22850 raft_consensus.cc:686] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [term 1 LEADER]: Becoming Leader. State: Replica: da545bfd50e24c45bf4f8fd92b67f3a4, State: Running, Role: LEADER
I20260927 10:53:51.392227 22850 consensus_queue.cc:237] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da545bfd50e24c45bf4f8fd92b67f3a4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40659 } }
I20260927 10:53:51.392730 22845 sys_catalog.cc:564] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:51.393472 22850 sys_catalog.cc:455] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da545bfd50e24c45bf4f8fd92b67f3a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da545bfd50e24c45bf4f8fd92b67f3a4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40659 } } }
I20260927 10:53:51.393680 22850 sys_catalog.cc:458] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:51.393915 22851 sys_catalog.cc:455] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da545bfd50e24c45bf4f8fd92b67f3a4. Latest consensus state: current_term: 1 leader_uuid: "da545bfd50e24c45bf4f8fd92b67f3a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da545bfd50e24c45bf4f8fd92b67f3a4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40659 } } }
I20260927 10:53:51.394091 22851 sys_catalog.cc:458] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:51.394136 22859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:51.396975 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:51.382864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38140 (local address 127.14.68.62:40659)
0927 10:53:51.383287 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:51.383301 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:51.383337 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:53:51.384253 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:51.384275 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:51.384610 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:51.384962 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:51.385784 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.386760 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.387656 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.388173 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.388342 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:51.391357 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:51.391383 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:51.391403 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:51.391441 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:51.393669 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:51.394233 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:51.394240 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:51.394248 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:51.394406 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:51.394757 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:51.394762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:51.394767 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:51.395269 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:51.395484 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:51.395797 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 10:53:51.396077 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":121,"threads_started":1}
I20260927 10:53:51.397347 22859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:51.402639 22859 catalog_manager.cc:1347] Generated new cluster ID: d8635c990014493d85e5322e4f665343
I20260927 10:53:51.402702 22859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:51.407842 22859 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:53:51.409029 22859 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:51.420447 22859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4: Generated new TSK 0
I20260927 10:53:51.421224 22859 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
I20260927 10:53:51.450407 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:40659
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:42709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:51.558753 22871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:51.558987 22871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:51.559027 22871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:51.563022 22871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:51.563084 22871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:51.563194 22871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:53:51.568292 22871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:42709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:51.569307 22871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:51.570309 22871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:51.579591 22876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:51.579634 22879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:51.579758 22871 server_base.cc:1034] running on GCE node
W20260927 10:53:51.579645 22877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:51.580219 22871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:51.580803 22871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:51.581986 22871 hybrid_clock.cc:648] HybridClock initialized: now 1790506431581957 us; error 43 us; skew 500 ppm
Sep 27 10:53:51 dist-test-slave-tz51 krb5kdc[22764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506431, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:51.584643 22871 init.cc:377] Logged in from keytab as kudu/127.14.68.1@KRBTEST.COM (short username kudu)
I20260927 10:53:51.585412 22871 webserver.cc:460] Webserver started at http://127.14.68.1:38675/ using document root <none> and password file <none>
I20260927 10:53:51.585963 22871 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:51.586024 22871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:51.586205 22871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:51.588174 22871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "c05901b6458f4a2b8179109ba669e2ba"
format_stamp: "Formatted at 2026-09-27 10:53:51 on dist-test-slave-tz51"
server_key: "05e3e50965fe02f5a944c62c26c0a5fa"
server_key_iv: "19eea98c177151ea734d7385de511d8c"
server_key_version: "encryptionkey@0"
I20260927 10:53:51.588670 22871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "c05901b6458f4a2b8179109ba669e2ba"
format_stamp: "Formatted at 2026-09-27 10:53:51 on dist-test-slave-tz51"
server_key: "05e3e50965fe02f5a944c62c26c0a5fa"
server_key_iv: "19eea98c177151ea734d7385de511d8c"
server_key_version: "encryptionkey@0"
I20260927 10:53:51.592142 22871 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:53:51.594538 22886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:51.595806 22871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:51.595898 22871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "c05901b6458f4a2b8179109ba669e2ba"
format_stamp: "Formatted at 2026-09-27 10:53:51 on dist-test-slave-tz51"
server_key: "05e3e50965fe02f5a944c62c26c0a5fa"
server_key_iv: "19eea98c177151ea734d7385de511d8c"
server_key_version: "encryptionkey@0"
I20260927 10:53:51.596043 22871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:51.620879 22871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:51.623654 22871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:51.623864 22871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:51.624399 22871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:51.625157 22871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:51.625211 22871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:51.625250 22871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:51.625265 22871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:51.633838 22871 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:43745
I20260927 10:53:51.633867 22999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:43745 every 8 connection(s)
I20260927 10:53:51.634725 22871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:53:51.636780 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 22871
I20260927 10:53:51.636925 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:53:51.637244 14608 external_mini_cluster.cc:1428] Setting key 2fc9cf234fd428df836eec060cea8fd0
Sep 27 10:53:51 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506431, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.68.2@KRBTEST.COM; defaulting to no policy
I20260927 10:53:51.649768 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:51.636570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:43951 (local address 127.14.68.62:40659)
0927 10:53:51.636757 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:51.636762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:51.637866 (+ 1104us) server_negotiation.cc:407] Connection header received
0927 10:53:51.639054 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:51.639058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:51.639113 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:51.639226 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:51.640441 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.640961 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.642101 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.642517 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.642634 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:51.645151 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:51.645158 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:51.645171 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:51.645200 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:51.647096 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:51.647818 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:51.647825 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:51.647829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:51.647934 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:51.648374 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:51.648377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:51.648378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:51.648558 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:51.648634 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:51.649396 (+ 762us) server_negotiation.cc:299] Negotiation successful
0927 10:53:51.649544 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.2@KRBTEST.COM" created.
I20260927 10:53:51.650920 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:51.636972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.1:43951)
0927 10:53:51.637712 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:51.637764 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:51.638800 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:51.639401 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:51.639408 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:51.639836 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:51.640301 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:51.640314 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.641094 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.641097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:51.641980 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:51.641989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.642674 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.642678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:51.642828 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:51.643465 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:51.643489 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:51.644984 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:51.647255 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:51.647264 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:51.647280 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:51.647634 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:51.648060 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:51.648066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:51.648069 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:51.648253 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:51.648678 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:51.648688 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:51.649060 (+ 372us) client_negotiation.cc:769] Sending connection context
0927 10:53:51.649295 (+ 235us) client_negotiation.cc:240] Negotiation successful
0927 10:53:51.649727 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":148,"threads_started":1}
I20260927 10:53:51.652405 23000 heartbeater.cc:347] Connected to a master server at 127.14.68.62:40659
I20260927 10:53:51.652666 23000 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:51.653262 23000 heartbeater.cc:511] Master 127.14.68.62:40659 requested a full tablet report, sending...
I20260927 10:53:51.654940 22809 ts_manager.cc:194] Registered new tserver with Master: c05901b6458f4a2b8179109ba669e2ba (127.14.68.1:43745)
I20260927 10:53:51.656607 22809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:43951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
I20260927 10:53:51.668947 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:40659
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:42709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:51.774863 23005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:51.775182 23005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:51.775266 23005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:51.779505 23005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:51.779584 23005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:51.779682 23005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:53:51.784739 23005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:42709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:51.785708 23005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:51.786715 23005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:51.796051 23013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:51.796077 23010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:51.796136 23005 server_base.cc:1034] running on GCE node
W20260927 10:53:51.796068 23011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:51.796646 23005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:51.797260 23005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:51.798439 23005 hybrid_clock.cc:648] HybridClock initialized: now 1790506431798426 us; error 24 us; skew 500 ppm
Sep 27 10:53:51 dist-test-slave-tz51 krb5kdc[22764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506431, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:51.801270 23005 init.cc:377] Logged in from keytab as kudu/127.14.68.2@KRBTEST.COM (short username kudu)
I20260927 10:53:51.801991 23005 webserver.cc:460] Webserver started at http://127.14.68.2:40665/ using document root <none> and password file <none>
I20260927 10:53:51.802453 23005 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:51.802506 23005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:51.802661 23005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:51.804641 23005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "edcb459997f14fc6af75573d345c3f69"
format_stamp: "Formatted at 2026-09-27 10:53:51 on dist-test-slave-tz51"
server_key: "558597e8babfcadae1257f8fdf73ada8"
server_key_iv: "d6f7d7954961cfbfed5afa8e5bac979f"
server_key_version: "encryptionkey@0"
I20260927 10:53:51.805153 23005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "edcb459997f14fc6af75573d345c3f69"
format_stamp: "Formatted at 2026-09-27 10:53:51 on dist-test-slave-tz51"
server_key: "558597e8babfcadae1257f8fdf73ada8"
server_key_iv: "d6f7d7954961cfbfed5afa8e5bac979f"
server_key_version: "encryptionkey@0"
I20260927 10:53:51.808514 23005 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:53:51.810761 23020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:51.811951 23005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:51.812062 23005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "edcb459997f14fc6af75573d345c3f69"
format_stamp: "Formatted at 2026-09-27 10:53:51 on dist-test-slave-tz51"
server_key: "558597e8babfcadae1257f8fdf73ada8"
server_key_iv: "d6f7d7954961cfbfed5afa8e5bac979f"
server_key_version: "encryptionkey@0"
I20260927 10:53:51.812162 23005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:51.824040 23005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:51.827083 23005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:51.827272 23005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:51.827868 23005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:51.828609 23005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:51.828660 23005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:51.828739 23005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:51.828776 23005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:51.837289 23005 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:39129
I20260927 10:53:51.837317 23133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:39129 every 8 connection(s)
I20260927 10:53:51.838068 23005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:53:51.844538 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 23005
I20260927 10:53:51.844676 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:53:51.844987 14608 external_mini_cluster.cc:1428] Setting key 7fafbdc29095e0f0cb0f55a5f5598782
Sep 27 10:53:51 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506431, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:51.851689 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:51.839804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:51197 (local address 127.14.68.62:40659)
0927 10:53:51.839924 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:51.839928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:51.840928 (+ 1000us) server_negotiation.cc:407] Connection header received
0927 10:53:51.841759 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:51.841761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:51.841808 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:51.841879 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:51.842933 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.843452 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.844768 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.844987 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.845112 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:51.847666 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:51.847673 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:51.847678 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:51.847734 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:51.849368 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:51.849933 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:51.849938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:51.849940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:51.850022 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:51.850496 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:51.850500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:51.850503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:51.850672 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:51.850747 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:51.851317 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 10:53:51.851452 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 10:53:51.852313 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:51.840154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.2:51197)
0927 10:53:51.840792 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:51.840829 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:51.841563 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:51.842007 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:51.842014 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:51.842349 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:51.842812 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:51.842825 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.843557 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.843559 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:51.844579 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:51.844590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:51.845164 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:51.845169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:51.845353 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:51.845987 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:51.846002 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:51.847500 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:51.849482 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:51.849488 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:51.849504 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:51.849834 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:51.850150 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:51.850163 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:51.850168 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:51.850367 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:51.850782 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:51.850789 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:51.851067 (+ 278us) client_negotiation.cc:769] Sending connection context
0927 10:53:51.851228 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 10:53:51.851460 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":106,"threads_started":1}
I20260927 10:53:51.853544 23134 heartbeater.cc:347] Connected to a master server at 127.14.68.62:40659
I20260927 10:53:51.853777 23134 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:51.854226 23134 heartbeater.cc:511] Master 127.14.68.62:40659 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.68.3@KRBTEST.COM; defaulting to no policy
I20260927 10:53:51.855194 22809 ts_manager.cc:194] Registered new tserver with Master: edcb459997f14fc6af75573d345c3f69 (127.14.68.2:39129)
I20260927 10:53:51.855862 22809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:51197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
I20260927 10:53:51.872522 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:40659
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:42709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:51.982760 23139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:51.982998 23139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:51.983042 23139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:51.987046 23139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:51.987105 23139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:51.987162 23139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:53:51.992319 23139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:42709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:51.993335 23139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:51.994385 23139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:52.003968 23147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:52.004009 23145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:52.003974 23144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:52.004166 23139 server_base.cc:1034] running on GCE node
I20260927 10:53:52.004514 23139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:52.005149 23139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:52.006335 23139 hybrid_clock.cc:648] HybridClock initialized: now 1790506432006314 us; error 44 us; skew 500 ppm
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506432, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:52.009038 23139 init.cc:377] Logged in from keytab as kudu/127.14.68.3@KRBTEST.COM (short username kudu)
I20260927 10:53:52.009855 23139 webserver.cc:460] Webserver started at http://127.14.68.3:40721/ using document root <none> and password file <none>
I20260927 10:53:52.010352 23139 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:52.010408 23139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:52.010563 23139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:53:52.012503 23139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "fe124f7a70c7486eb3ae0e070db1bd45"
format_stamp: "Formatted at 2026-09-27 10:53:52 on dist-test-slave-tz51"
server_key: "f026765747afe438fc8b833941acee3c"
server_key_iv: "39a4c13360f74c57dfc901055f4e0812"
server_key_version: "encryptionkey@0"
I20260927 10:53:52.013023 23139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "fe124f7a70c7486eb3ae0e070db1bd45"
format_stamp: "Formatted at 2026-09-27 10:53:52 on dist-test-slave-tz51"
server_key: "f026765747afe438fc8b833941acee3c"
server_key_iv: "39a4c13360f74c57dfc901055f4e0812"
server_key_version: "encryptionkey@0"
I20260927 10:53:52.016734 23139 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:53:52.019091 23154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:52.020449 23139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:53:52.020545 23139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "fe124f7a70c7486eb3ae0e070db1bd45"
format_stamp: "Formatted at 2026-09-27 10:53:52 on dist-test-slave-tz51"
server_key: "f026765747afe438fc8b833941acee3c"
server_key_iv: "39a4c13360f74c57dfc901055f4e0812"
server_key_version: "encryptionkey@0"
I20260927 10:53:52.020627 23139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:52.043447 23139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:52.046241 23139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:52.046391 23139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:52.046885 23139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:52.047627 23139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:52.047672 23139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:52.047731 23139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:52.047755 23139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:52.056982 23139 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:34207
I20260927 10:53:52.057077 23267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:34207 every 8 connection(s)
I20260927 10:53:52.057878 23139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:53:52.058858 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 23139
I20260927 10:53:52.058984 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:53:52.059265 14608 external_mini_cluster.cc:1428] Setting key da0c5c7d6d85ce12d6a1a9136b86c416
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506432, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:53:52.070886 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:52.059791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:36321 (local address 127.14.68.62:40659)
0927 10:53:52.059947 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:52.059951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:52.061028 (+ 1077us) server_negotiation.cc:407] Connection header received
0927 10:53:52.061972 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:52.061975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:52.062026 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:52.062099 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:52.063208 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.063863 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.064980 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.065218 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.065329 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:52.067621 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:52.067624 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:52.067626 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:52.067650 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:52.068915 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:52.069403 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:52.069408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:52.069410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:52.069494 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:52.069846 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:52.069849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:52.069851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:52.070031 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:52.070139 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:52.070596 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 10:53:52.070719 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 10:53:52.071604 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:52.060235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.3:36321)
0927 10:53:52.060893 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:52.060932 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:52.061785 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:52.062266 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:52.062272 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:52.062617 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:52.063062 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:52.063075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.063984 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.063988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:52.064853 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:52.064862 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.065361 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.065365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:52.065539 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:52.066034 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:52.066050 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:52.067448 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:53:52.069028 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:52.069033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:52.069047 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:52.069295 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:52.069610 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:53:52.069613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:53:52.069615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:53:52.069745 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:53:52.070134 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:53:52.070141 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:53:52.070373 (+ 232us) client_negotiation.cc:769] Sending connection context
0927 10:53:52.070554 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 10:53:52.070849 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":106,"threads_started":1}
I20260927 10:53:52.072969 23268 heartbeater.cc:347] Connected to a master server at 127.14.68.62:40659
I20260927 10:53:52.073220 23268 heartbeater.cc:464] Registering TS with master...
I20260927 10:53:52.073711 23268 heartbeater.cc:511] Master 127.14.68.62:40659 requested a full tablet report, sending...
I20260927 10:53:52.074669 22808 ts_manager.cc:194] Registered new tserver with Master: fe124f7a70c7486eb3ae0e070db1bd45 (127.14.68.3:34207)
I20260927 10:53:52.075208 22808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:36321
I20260927 10:53:52.085228 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.1.keytab.
Entry for principal oryx/127.14.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.1.keytab.
WARNING: no policy specified for oryx/127.14.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.68.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.2.keytab.
Entry for principal oryx/127.14.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.2.keytab.
WARNING: no policy specified for oryx/127.14.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.3.keytab.
Entry for principal oryx/127.14.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.3.keytab.
I20260927 10:53:52.154352 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 22871
I20260927 10:53:52.160079 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 23005
I20260927 10:53:52.165680 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 23139
I20260927 10:53:52.170718 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 22777
I20260927 10:53:52.176847 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:40659
--webserver_interface=127.14.68.62
--webserver_port=34123
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:42709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:40659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:53:52.284741 23277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:52.284983 23277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:52.285020 23277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:52.289058 23277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:53:52.289111 23277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:52.289130 23277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:52.289214 23277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:53:52.289244 23277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:53:52.294413 23277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:42709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:40659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:40659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=34123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:52.295409 23277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:52.296420 23277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:52.304662 23282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:52.304776 23277 server_base.cc:1034] running on GCE node
W20260927 10:53:52.304683 23283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:52.304690 23285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:52.305440 23277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:52.306507 23277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:52.307735 23277 hybrid_clock.cc:648] HybridClock initialized: now 1790506432307716 us; error 27 us; skew 500 ppm
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506432, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:52.310293 23277 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:53:52.311133 23277 webserver.cc:460] Webserver started at http://127.14.68.62:34123/ using document root <none> and password file <none>
I20260927 10:53:52.311685 23277 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:52.311851 23277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:52.315469 23277 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 10:53:52.317485 23292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:52.318611 23277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:53:52.318770 23277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "da545bfd50e24c45bf4f8fd92b67f3a4"
format_stamp: "Formatted at 2026-09-27 10:53:51 on dist-test-slave-tz51"
server_key: "68d5ddd00ca2c8aaad31080a167acbb0"
server_key_iv: "8a73a47f5ba5b5a731ec446f4b2dacc9"
server_key_version: "encryptionkey@0"
I20260927 10:53:52.319284 23277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:52.341892 23277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:52.344861 23277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:52.345067 23277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:52.351972 23277 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:40659
I20260927 10:53:52.351998 23344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:40659 every 8 connection(s)
I20260927 10:53:52.353063 23277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:53:52.357143 23345 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:53:52.358727 23345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4: Bootstrap starting.
I20260927 10:53:52.362629 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 23277
I20260927 10:53:52.363133 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:43745
--local_ip_for_outbound_sockets=127.14.68.1
--tserver_master_addrs=127.14.68.62:40659
--webserver_port=38675
--webserver_interface=127.14.68.1
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:42709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.68.1
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:53:52.375324 23345 log.cc:824] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4: Log is configured to *not* fsync() on all Append() calls
I20260927 10:53:52.382097 23345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:53:52.382489 23345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4: Bootstrap complete.
I20260927 10:53:52.386679 23345 raft_consensus.cc:348] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da545bfd50e24c45bf4f8fd92b67f3a4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40659 } }
I20260927 10:53:52.387102 23345 raft_consensus.cc:729] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: da545bfd50e24c45bf4f8fd92b67f3a4, State: Initialized, Role: FOLLOWER
I20260927 10:53:52.387578 23345 consensus_queue.cc:260] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da545bfd50e24c45bf4f8fd92b67f3a4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40659 } }
I20260927 10:53:52.387745 23345 raft_consensus.cc:388] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:53:52.387837 23345 raft_consensus.cc:482] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:53:52.387959 23345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:53:52.389842 23345 raft_consensus.cc:504] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da545bfd50e24c45bf4f8fd92b67f3a4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40659 } }
I20260927 10:53:52.390183 23345 leader_election.cc:302] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da545bfd50e24c45bf4f8fd92b67f3a4; no voters:
I20260927 10:53:52.390535 23345 leader_election.cc:288] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 10:53:52.390620 23349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 10:53:52.390815 23349 raft_consensus.cc:686] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [term 2 LEADER]: Becoming Leader. State: Replica: da545bfd50e24c45bf4f8fd92b67f3a4, State: Running, Role: LEADER
I20260927 10:53:52.391104 23349 consensus_queue.cc:237] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da545bfd50e24c45bf4f8fd92b67f3a4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40659 } }
I20260927 10:53:52.391775 23345 sys_catalog.cc:564] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:53:52.392428 23350 sys_catalog.cc:455] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "da545bfd50e24c45bf4f8fd92b67f3a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da545bfd50e24c45bf4f8fd92b67f3a4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40659 } } }
I20260927 10:53:52.392553 23350 sys_catalog.cc:458] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:52.393184 23351 sys_catalog.cc:455] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da545bfd50e24c45bf4f8fd92b67f3a4. Latest consensus state: current_term: 2 leader_uuid: "da545bfd50e24c45bf4f8fd92b67f3a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da545bfd50e24c45bf4f8fd92b67f3a4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40659 } } }
I20260927 10:53:52.393332 23351 sys_catalog.cc:458] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4 [sys.catalog]: This master's current role is: LEADER
I20260927 10:53:52.393620 23358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:53:52.397156 23358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:53:52.397779 23358 catalog_manager.cc:1259] Loaded cluster ID: d8635c990014493d85e5322e4f665343
I20260927 10:53:52.397842 23358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:53:52.400270 23358 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:53:52.400911 23358 catalog_manager.cc:5879] T 00000000000000000000000000000000 P da545bfd50e24c45bf4f8fd92b67f3a4: Loaded TSK: 0
I20260927 10:53:52.402112 23358 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:53:52.477505 23347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:52.477753 23347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:52.477797 23347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:52.481927 23347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:52.481983 23347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:52.482038 23347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:53:52.487753 23347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:42709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.1.keytab
--principal=oryx/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:43745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=38675
--tserver_master_addrs=127.14.68.62:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:52.488686 23347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:52.489610 23347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:52.500087 23371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:52.500247 23347 server_base.cc:1034] running on GCE node
W20260927 10:53:52.500072 23372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:52.500083 23374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:52.500842 23347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:52.501435 23347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:52.502621 23347 hybrid_clock.cc:648] HybridClock initialized: now 1790506432502596 us; error 45 us; skew 500 ppm
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506432, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:52.505412 23347 init.cc:377] Logged in from keytab as oryx/127.14.68.1@KRBTEST.COM (short username oryx)
I20260927 10:53:52.506337 23347 webserver.cc:460] Webserver started at http://127.14.68.1:38675/ using document root <none> and password file <none>
I20260927 10:53:52.506863 23347 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:52.506961 23347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:52.510797 23347 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:52.512954 23381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:52.514232 23347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:53:52.514351 23347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "c05901b6458f4a2b8179109ba669e2ba"
format_stamp: "Formatted at 2026-09-27 10:53:51 on dist-test-slave-tz51"
server_key: "05e3e50965fe02f5a944c62c26c0a5fa"
server_key_iv: "19eea98c177151ea734d7385de511d8c"
server_key_version: "encryptionkey@0"
I20260927 10:53:52.514849 23347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:52.536917 23347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:52.541647 23347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:52.541846 23347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:52.542456 23347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:52.543290 23347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:52.543356 23347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:52.543416 23347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:52.543449 23347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:52.556699 23347 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:43745
I20260927 10:53:52.556757 23494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:43745 every 8 connection(s)
I20260927 10:53:52.558064 23347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:53:52.559713 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 23347
I20260927 10:53:52.560276 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:39129
--local_ip_for_outbound_sockets=127.14.68.2
--tserver_master_addrs=127.14.68.62:40659
--webserver_port=40665
--webserver_interface=127.14.68.2
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:42709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.68.2
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:52.573872 23497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:52.560859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.1:49885)
0927 10:53:52.561647 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:52.561699 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:52.562959 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:52.564992 (+ 2033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:52.565005 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:52.565581 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:52.566196 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:52.566227 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.567645 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.567651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:52.568617 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:52.568626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.569433 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.569437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:52.569627 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:52.570224 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:52.570239 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:52.572900 (+ 2661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":137,"threads_started":1}
W20260927 10:53:52.574178 23499 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:52.561578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:49885 (local address 127.14.68.62:40659)
0927 10:53:52.562047 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:52.562069 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:52.562116 (+ 47us) server_negotiation.cc:407] Connection header received
0927 10:53:52.563577 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:52.563596 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:52.564053 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:52.564425 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:52.566384 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.567496 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.568741 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.569251 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.569428 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:52.573130 (+ 3702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.1:49885: BlockingRecv error: recv got EOF from 127.14.68.1:49885 (error 108)
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":125,"threads_started":1}
W20260927 10:53:52.574796 23495 heartbeater.cc:644] Failed to heartbeat to 127.14.68.62:40659 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.68.62:40659: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:52.580607 23497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:52.575246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.1:36567)
0927 10:53:52.575360 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:52.575374 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:52.575448 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:52.575667 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:52.575670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:52.575851 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:52.575933 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:52.575938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.576750 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.576757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:52.577443 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:52.577452 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.578076 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.578080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:52.578153 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:52.578669 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:52.578720 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:52.580517 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":32}
W20260927 10:53:52.580799 23499 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:52.575339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:36567 (local address 127.14.68.62:40659)
0927 10:53:52.575454 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:52.575459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:52.575474 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:53:52.575513 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:52.575517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:52.575565 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:52.575654 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:52.576044 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.576650 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.577574 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.577952 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.578081 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:52.580660 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.1:36567: BlockingRecv error: recv got EOF from 127.14.68.1:36567 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:52.587028 23497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:52.581151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.1:53461)
0927 10:53:52.581232 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:52.581245 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:52.581326 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:52.581550 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:52.581553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:52.581705 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:52.581764 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:52.581769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.582489 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.582492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:52.583446 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:52.583459 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.584383 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.584389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:52.584470 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:52.584923 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:52.584936 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:52.586917 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 10:53:52.587218 23495 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:53:52.587257 23499 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:52.581182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:53461 (local address 127.14.68.62:40659)
0927 10:53:52.581269 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:52.581272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:52.581305 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:53:52.581405 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:52.581408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:52.581446 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:52.581538 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:52.581870 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.582343 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.583590 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.584258 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.584367 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:52.587155 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.1:53461: BlockingRecv error: recv got EOF from 127.14.68.1:53461 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 10:53:52.679515 23498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:52.679790 23498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:52.679832 23498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:52.683760 23498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:52.683808 23498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:52.683911 23498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:53:52.688982 23498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:42709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.2.keytab
--principal=oryx/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:39129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=40665
--tserver_master_addrs=127.14.68.62:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:52.689999 23498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:52.690985 23498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:52.701046 23504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:52.701125 23507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:52.701046 23505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:52.701097 23498 server_base.cc:1034] running on GCE node
I20260927 10:53:52.701743 23498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:52.702338 23498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:52.703496 23498 hybrid_clock.cc:648] HybridClock initialized: now 1790506432703484 us; error 25 us; skew 500 ppm
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506432, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:52.706149 23498 init.cc:377] Logged in from keytab as oryx/127.14.68.2@KRBTEST.COM (short username oryx)
I20260927 10:53:52.706912 23498 webserver.cc:460] Webserver started at http://127.14.68.2:40665/ using document root <none> and password file <none>
I20260927 10:53:52.707397 23498 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:52.707463 23498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:52.711004 23498 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:53:52.712930 23514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:52.714047 23498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:53:52.714164 23498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "edcb459997f14fc6af75573d345c3f69"
format_stamp: "Formatted at 2026-09-27 10:53:51 on dist-test-slave-tz51"
server_key: "558597e8babfcadae1257f8fdf73ada8"
server_key_iv: "d6f7d7954961cfbfed5afa8e5bac979f"
server_key_version: "encryptionkey@0"
I20260927 10:53:52.714669 23498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:52.736791 23498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:52.740048 23498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:52.740255 23498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:52.740835 23498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:52.741619 23498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:52.741671 23498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:52.741746 23498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:52.741783 23498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:52.750397 23498 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:39129
I20260927 10:53:52.750428 23627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:39129 every 8 connection(s)
I20260927 10:53:52.751330 23498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:53:52.757463 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 23498
I20260927 10:53:52.757946 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:34207
--local_ip_for_outbound_sockets=127.14.68.3
--tserver_master_addrs=127.14.68.62:40659
--webserver_port=40721
--webserver_interface=127.14.68.3
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:42709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.68.3
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 10:53:52.761901 23499 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:52.752943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:55785 (local address 127.14.68.62:40659)
0927 10:53:52.753098 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:52.753103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:52.754066 (+ 963us) server_negotiation.cc:407] Connection header received
0927 10:53:52.754869 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:52.754871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:52.754922 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:52.754995 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:52.756103 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.756626 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.757788 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.758147 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.758265 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:52.761777 (+ 3512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.2:55785: BlockingRecv error: recv got EOF from 127.14.68.2:55785 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:53:52.762725 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:52.753284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.2:55785)
0927 10:53:52.753922 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:52.753962 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:52.754701 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:52.755164 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:52.755170 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:52.755503 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:52.755983 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:52.755996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.756726 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.756728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:52.757673 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:52.757683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.758302 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.758305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:52.758435 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:52.758953 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:52.758968 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:52.761678 (+ 2710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":111,"threads_started":1}
W20260927 10:53:52.763149 23628 heartbeater.cc:644] Failed to heartbeat to 127.14.68.62:40659 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.68.62:40659: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:52.768443 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:52.763565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.2:57299)
0927 10:53:52.763652 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:52.763673 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:52.763785 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:52.764000 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:52.764002 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:52.764149 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:52.764241 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:52.764246 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.764949 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.764952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:52.765516 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:52.765523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.765949 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.765952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:52.766043 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:52.766495 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:52.766506 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:52.768285 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 10:53:52.768457 23499 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:52.763664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:57299 (local address 127.14.68.62:40659)
0927 10:53:52.763802 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:52.763808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:52.763824 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:53:52.763862 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:52.763865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:52.763907 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:52.763987 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:52.764346 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.764825 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.765632 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.765844 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.765949 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:52.768380 (+ 2431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.2:57299: BlockingRecv error: recv got EOF from 127.14.68.2:57299 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:52.773453 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:52.768987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.2:49685)
0927 10:53:52.769088 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:52.769104 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:52.769171 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:52.769385 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:52.769388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:52.769526 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:52.769586 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:52.769590 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.770285 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.770290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:52.770782 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:52.770789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.771222 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.771226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:52.771319 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:52.771762 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:52.771773 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:52.773334 (+ 1561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 10:53:52.773525 23499 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:52.769045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:49685 (local address 127.14.68.62:40659)
0927 10:53:52.769171 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:52.769176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:52.769189 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:53:52.769250 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:52.769253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:52.769293 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:52.769372 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:52.769710 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.770180 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.770897 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.771103 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.771223 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:52.773427 (+ 2204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.2:49685: BlockingRecv error: recv got EOF from 127.14.68.2:49685 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260927 10:53:52.773768 23628 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:53:52.869519 23631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:53:52.869735 23631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:53:52.869794 23631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:53:52.873605 23631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:53:52.873649 23631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:53:52.873703 23631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:53:52.879951 23631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:42709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/krb5kdc/oryx_127.14.68.3.keytab
--principal=oryx/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:34207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=40721
--tserver_master_addrs=127.14.68.62:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:53:52.880832 23631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:53:52.881734 23631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:53:52.891002 23639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:52.891007 23636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:53:52.891002 23637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:53:52.891259 23631 server_base.cc:1034] running on GCE node
I20260927 10:53:52.891855 23631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:53:52.892462 23631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:53:52.893646 23631 hybrid_clock.cc:648] HybridClock initialized: now 1790506432893596 us; error 62 us; skew 500 ppm
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506432, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:53:52.896275 23631 init.cc:377] Logged in from keytab as oryx/127.14.68.3@KRBTEST.COM (short username oryx)
I20260927 10:53:52.897018 23631 webserver.cc:460] Webserver started at http://127.14.68.3:40721/ using document root <none> and password file <none>
I20260927 10:53:52.897670 23631 fs_manager.cc:359] Metadata directory not provided
I20260927 10:53:52.897743 23631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:53:52.901363 23631 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260927 10:53:52.903369 23646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:52.904577 23631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:53:52.904702 23631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "fe124f7a70c7486eb3ae0e070db1bd45"
format_stamp: "Formatted at 2026-09-27 10:53:52 on dist-test-slave-tz51"
server_key: "f026765747afe438fc8b833941acee3c"
server_key_iv: "39a4c13360f74c57dfc901055f4e0812"
server_key_version: "encryptionkey@0"
I20260927 10:53:52.905217 23631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:53:52.929951 23631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:53:52.933147 23631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:53:52.933349 23631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:53:52.933945 23631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:53:52.934767 23631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:53:52.934811 23631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:52.934882 23631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:53:52.934922 23631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:53:52.943538 23631 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:34207
I20260927 10:53:52.943607 23759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:34207 every 8 connection(s)
I20260927 10:53:52.944526 23631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:52.954718 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 23631
W20260927 10:53:52.955029 23499 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:52.946237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:57239 (local address 127.14.68.62:40659)
0927 10:53:52.946364 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:52.946367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:52.947304 (+ 937us) server_negotiation.cc:407] Connection header received
0927 10:53:52.948287 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:52.948290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:52.948349 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:52.948450 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:52.949524 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.950042 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.951240 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.951583 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.951689 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:52.954933 (+ 3244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.3:57239: BlockingRecv error: recv got EOF from 127.14.68.3:57239 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:53:52.955822 23762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:52.946538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.3:57239)
0927 10:53:52.947155 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:52.947194 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:52.948064 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:52.948617 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:52.948624 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:52.948945 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:52.949383 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:52.949396 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.950162 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.950165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:52.951107 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:52.951117 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.951751 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.951756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:52.951918 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:52.952460 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:52.952475 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:52.954795 (+ 2320us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":113,"threads_started":1}
W20260927 10:53:52.956283 23760 heartbeater.cc:644] Failed to heartbeat to 127.14.68.62:40659 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.68.62:40659: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:52.961786 23762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:52.956706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.3:37781)
0927 10:53:52.956864 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:52.956879 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:52.956957 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:52.957181 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:52.957184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:52.957351 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:52.957480 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:52.957485 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.958211 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.958218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:52.958757 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:52.958765 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.959239 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.959242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:52.959330 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:52.959907 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:52.959918 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:52.961637 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94}
W20260927 10:53:52.962221 23764 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:52.956769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:37781 (local address 127.14.68.62:40659)
0927 10:53:52.956955 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:52.956957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:52.956968 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:53:52.957038 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:52.957040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:52.957070 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:52.957136 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:52.957594 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.958098 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.958874 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.959098 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.959227 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:52.961784 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.3:37781: BlockingRecv error: recv got EOF from 127.14.68.3:37781 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":71,"threads_started":1}
I20260927 10:53:52.965056 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:52.955761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56576 (local address 127.14.68.62:40659)
0927 10:53:52.955905 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:52.955909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:52.955924 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:53:52.955967 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:52.955972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:52.956009 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:52.956086 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:52.957362 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.957913 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.958776 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.958997 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.959098 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:52.960583 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:53:52.960589 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:53:52.960602 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:53:52.960628 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:53:52.962978 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:52.963391 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:52.963397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:52.963403 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:52.963544 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:53:52.963843 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:53:52.963848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:53:52.963850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:53:52.964209 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:53:52.964351 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 10:53:52.964675 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 10:53:52.964895 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:52 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:52.967418 23762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:52.962328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.3:37751)
0927 10:53:52.962481 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:52.962495 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:52.962571 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:52.962822 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:52.962825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:52.962938 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:52.963000 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:52.963004 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.963905 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.963907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:52.964519 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:52.964528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.965045 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.965048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:52.965106 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:52.965513 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:52.965526 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:52.967291 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84}
W20260927 10:53:52.967516 23764 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:52.962371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:37751 (local address 127.14.68.62:40659)
0927 10:53:52.962494 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:52.962498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:52.962562 (+ 64us) server_negotiation.cc:407] Connection header received
0927 10:53:52.962660 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:52.962669 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:52.962716 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:52.962804 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:52.963124 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.963796 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.964618 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:52.964930 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:52.965023 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:52.967390 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.3:37751: BlockingRecv error: recv got EOF from 127.14.68.3:37751 (error 108)
Metrics: {"server-negotiator.queue_time_us":69,"thread_start_us":59,"threads_started":1}
W20260927 10:53:52.967728 23760 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 10:53:53 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:53 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:53.593396 23766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:53.587912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.1:55835)
0927 10:53:53.588217 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:53.588253 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:53.588370 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:53.588613 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:53.588615 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:53.588790 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:53.588902 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:53.588907 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:53.589605 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:53.589607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:53.590179 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:53.590186 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:53.590578 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:53.590580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:53.590653 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:53.591110 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:53.591120 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:53.593193 (+ 2073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":82,"threads_started":1}
W20260927 10:53:53.593564 23767 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:53.587975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:55835 (local address 127.14.68.62:40659)
0927 10:53:53.588232 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:53.588239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:53.588309 (+ 70us) server_negotiation.cc:407] Connection header received
0927 10:53:53.588457 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:53.588459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:53.588503 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:53.588577 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:53.588998 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:53.589517 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:53.590275 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:53.590486 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:53.590558 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:53.593335 (+ 2777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.1:55835: BlockingRecv error: recv got EOF from 127.14.68.1:55835 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
Sep 27 10:53:53 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:53 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:53.779417 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:53.774301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.2:51751)
0927 10:53:53.774564 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:53.774578 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:53.774680 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:53.774899 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:53.774902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:53.775069 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:53.775182 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:53.775186 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:53.776018 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:53.776021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:53.776585 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:53.776591 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:53.777002 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:53.777004 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:53.777073 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:53.777532 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:53.777542 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:53.779282 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":66,"threads_started":1}
W20260927 10:53:53.779558 23767 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:53.774389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:51751 (local address 127.14.68.62:40659)
0927 10:53:53.774573 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:53.774577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:53.774649 (+ 72us) server_negotiation.cc:407] Connection header received
0927 10:53:53.774759 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:53.774761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:53.774803 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:53.774864 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:53.775271 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:53.775926 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:53.776700 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:53.776908 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:53.776982 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:53.779429 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.2:51751: BlockingRecv error: recv got EOF from 127.14.68.2:51751 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 10:53:53 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:53 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:53.973388 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:53.968259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.3:59191)
0927 10:53:53.968531 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:53.968545 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:53.968640 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:53.968873 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:53.968875 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:53.969044 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:53.969136 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:53.969142 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:53.969905 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:53.969909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:53.970496 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:53.970505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:53.970928 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:53.970931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:53.970998 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:53.971463 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:53.971474 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:53.973270 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
W20260927 10:53:53.973553 23767 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:53.968403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:59191 (local address 127.14.68.62:40659)
0927 10:53:53.968550 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:53.968553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:53.968596 (+ 43us) server_negotiation.cc:407] Connection header received
0927 10:53:53.968727 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:53.968729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:53.968773 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:53.968836 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:53.969242 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:53.969778 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:53.970605 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:53.970816 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:53.970889 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:53.973403 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.3:59191: BlockingRecv error: recv got EOF from 127.14.68.3:59191 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 10:53:54 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:54 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:54.599478 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:54.594142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.1:44173)
0927 10:53:54.594406 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:54.594419 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:54.594518 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:54.594767 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:54.594769 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:54.594936 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:54.595027 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:54.595031 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:54.595785 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:54.595788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:54.596448 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:54.596457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:54.596911 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:54.596915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:54.597011 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:54.597547 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:54.597558 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:54.599362 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":84,"threads_started":1}
W20260927 10:53:54.599613 23771 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:54.594195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:44173 (local address 127.14.68.62:40659)
0927 10:53:54.594428 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:54.594432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:54.594487 (+ 55us) server_negotiation.cc:407] Connection header received
0927 10:53:54.594601 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:54.594603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:54.594648 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:54.594732 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:54.595121 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:54.595649 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:54.596563 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:54.596793 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:54.596908 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:54.599486 (+ 2578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.1:44173: BlockingRecv error: recv got EOF from 127.14.68.1:44173 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":60,"threads_started":1}
Sep 27 10:53:54 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:54 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:54.785523 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:54.780235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.2:54213)
0927 10:53:54.780497 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:54.780510 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:54.780638 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:54.780906 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:54.780909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:54.781109 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:54.781221 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:54.781227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:54.782039 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:54.782044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:54.782607 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:54.782614 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:54.783095 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:54.783099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:54.783192 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:54.783661 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:54.783674 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:54.785405 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":91,"threads_started":1}
W20260927 10:53:54.785682 23771 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:54.780306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:54213 (local address 127.14.68.62:40659)
0927 10:53:54.780452 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:54.780458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:54.780590 (+ 132us) server_negotiation.cc:407] Connection header received
0927 10:53:54.780750 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:54.780753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:54.780803 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:54.780884 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:54.781340 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:54.781931 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:54.782729 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:54.782987 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:54.783094 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:54.785529 (+ 2435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.2:54213: BlockingRecv error: recv got EOF from 127.14.68.2:54213 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 10:53:54 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:54 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:54.979903 23773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:54.974271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.3:43757)
0927 10:53:54.974684 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:54.974701 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:54.974850 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:54.975120 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:54.975122 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:54.975357 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:54.975447 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:54.975452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:54.976370 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:54.976376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:54.976923 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:54.976931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:54.977379 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:54.977383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:54.977465 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:54.978001 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:54.978012 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:54.979698 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":103,"threads_started":1}
W20260927 10:53:54.979965 23771 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:54.974438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:43757 (local address 127.14.68.62:40659)
0927 10:53:54.974698 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:54.974703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:54.974789 (+ 86us) server_negotiation.cc:407] Connection header received
0927 10:53:54.974951 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:54.974954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:54.975008 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:54.975098 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:54.975562 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:54.976263 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:54.977040 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:54.977273 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:54.977376 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:54.979853 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.3:43757: BlockingRecv error: recv got EOF from 127.14.68.3:43757 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
Sep 27 10:53:55 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:55 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:55.605892 23774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:55.600232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.1:56705)
0927 10:53:55.600544 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:55.600558 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:55.600662 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:55.600907 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:55.600909 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:55.601125 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:55.601222 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:55.601227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:55.601955 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:55.601958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:55.602568 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:55.602579 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:55.602982 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:55.602984 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:55.603053 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:55.603564 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:55.603575 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:55.605727 (+ 2152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":109,"threads_started":1}
W20260927 10:53:55.606155 23775 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:55.600374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:56705 (local address 127.14.68.62:40659)
0927 10:53:55.600590 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:55.600593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:55.600606 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:53:55.600756 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:55.600759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:55.600802 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:55.600873 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:55.601320 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:55.601834 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:55.602674 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:55.602891 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:55.602962 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:55.605862 (+ 2900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.1:56705: BlockingRecv error: recv got EOF from 127.14.68.1:56705 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":71,"threads_started":1}
Sep 27 10:53:55 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:55 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:55.791671 23776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:55.786230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.2:49469)
0927 10:53:55.786481 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:55.786499 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:55.786595 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:55.786829 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:55.786831 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:55.787006 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:55.787100 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:55.787105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:55.787935 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:55.787938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:55.788559 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:55.788571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:55.789091 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:55.789095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:55.789211 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:55.789760 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:55.789772 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:55.791556 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":85,"threads_started":1}
W20260927 10:53:55.792004 23775 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:55.786301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:49469 (local address 127.14.68.62:40659)
0927 10:53:55.786448 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:55.786452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:55.786567 (+ 115us) server_negotiation.cc:407] Connection header received
0927 10:53:55.786676 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:55.786678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:55.786722 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:55.786799 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:55.787231 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:55.787837 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:55.788708 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:55.788968 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:55.789084 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:55.791874 (+ 2790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.2:49469: BlockingRecv error: recv got EOF from 127.14.68.2:49469 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 10:53:55 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:55 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:55.986554 23777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:55.980674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.3:47589)
0927 10:53:55.980945 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:55.980958 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:55.981056 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:55.981295 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:55.981297 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:55.981472 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:55.981567 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:55.981572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:55.982276 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:55.982279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:55.982844 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:55.982851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:55.983404 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:55.983407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:55.983502 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:55.984386 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:55.984401 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:55.986359 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":102,"threads_started":1}
W20260927 10:53:55.986581 23775 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:55.980756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:47589 (local address 127.14.68.62:40659)
0927 10:53:55.980885 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:55.980888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:55.981023 (+ 135us) server_negotiation.cc:407] Connection header received
0927 10:53:55.981145 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:55.981147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:55.981194 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:55.981256 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:55.981661 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:55.982183 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:55.982936 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:55.983262 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:55.983381 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:55.986480 (+ 3099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.3:47589: BlockingRecv error: recv got EOF from 127.14.68.3:47589 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 10:53:56 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:56 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:56.615885 23778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:56.606707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.1:37963)
0927 10:53:56.606970 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:56.606983 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:56.607071 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:56.607389 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:56.607392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:56.607566 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:56.607658 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:56.607662 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:56.608694 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:56.608699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:56.609432 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:56.609449 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:56.609917 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:56.609921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:56.610008 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:56.613824 (+ 3816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:56.613836 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:56.615656 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":96,"threads_started":1}
W20260927 10:53:56.616041 23779 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:56.606898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:37963 (local address 127.14.68.62:40659)
0927 10:53:56.607179 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:56.607182 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:56.607194 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:53:56.607229 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:56.607231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:56.607272 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:56.607348 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:56.607826 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:56.608577 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:56.609558 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:56.609798 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:56.609911 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:56.615888 (+ 5977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.1:37963: BlockingRecv error: recv got EOF from 127.14.68.1:37963 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":118,"threads_started":1}
Sep 27 10:53:56 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:56 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:56.798133 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:56.792614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.2:35229)
0927 10:53:56.793076 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:56.793091 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:56.793231 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:56.793563 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:56.793566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:56.793771 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:56.793865 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:56.793871 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:56.794654 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:56.794656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:56.795227 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:56.795235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:56.795622 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:56.795625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:56.795687 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:56.796266 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:56.796277 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:56.798021 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":80,"threads_started":1}
W20260927 10:53:56.798323 23779 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:56.792739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:35229 (local address 127.14.68.62:40659)
0927 10:53:56.792879 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:56.792884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:56.793175 (+ 291us) server_negotiation.cc:407] Connection header received
0927 10:53:56.793348 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:56.793354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:56.793415 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:56.793517 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:56.794005 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:56.794559 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:56.795324 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:56.795531 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:56.795604 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:56.798150 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.2:35229: BlockingRecv error: recv got EOF from 127.14.68.2:35229 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 27 10:53:56 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:56 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:56.993060 23781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:56.987316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.3:60567)
0927 10:53:56.987651 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:56.987679 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:56.987816 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:56.988071 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:56.988073 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:56.988253 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:56.988346 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:56.988350 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:56.989050 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:56.989052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:56.989639 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:56.989650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:56.990254 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:56.990258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:56.990346 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:56.990918 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:56.990930 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:56.992894 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":88,"threads_started":1}
W20260927 10:53:56.993104 23779 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:56.987401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:60567 (local address 127.14.68.62:40659)
0927 10:53:56.987656 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:56.987661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:56.987754 (+ 93us) server_negotiation.cc:407] Connection header received
0927 10:53:56.987908 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:56.987911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:56.987962 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:56.988029 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:56.988444 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:56.988963 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:56.989763 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:56.990081 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:56.990240 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:56.993006 (+ 2766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.3:60567: BlockingRecv error: recv got EOF from 127.14.68.3:60567 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
Sep 27 10:53:57 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:57 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:57.622375 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:57.616707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.1:51119)
0927 10:53:57.617082 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:57.617096 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:57.617195 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:57.617454 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:57.617457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:57.617697 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:57.617787 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:57.617794 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:57.618617 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:57.618623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:57.619227 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:57.619235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:57.619741 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:57.619745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:57.619831 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:57.620391 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:57.620412 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:57.622189 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":86,"threads_started":1}
W20260927 10:53:57.622438 23783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:57.616775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:51119 (local address 127.14.68.62:40659)
0927 10:53:57.617054 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:57.617059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:57.617162 (+ 103us) server_negotiation.cc:407] Connection header received
0927 10:53:57.617269 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:57.617271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:57.617314 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:57.617430 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:57.617903 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:57.618506 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:57.619345 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:57.619577 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:57.619729 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:57.622308 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.1:51119: BlockingRecv error: recv got EOF from 127.14.68.1:51119 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":86,"threads_started":1}
Sep 27 10:53:57 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:57 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:57.804787 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:57.798895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.2:44857)
0927 10:53:57.799213 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:57.799231 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:57.799367 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:57.799632 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:57.799635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:57.799876 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:57.799972 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:57.799978 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:57.800764 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:57.800771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:57.801447 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:57.801480 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:57.802006 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:57.802010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:57.802099 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:57.802644 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:57.802660 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:57.804582 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":106,"threads_started":1}
W20260927 10:53:57.804824 23783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:57.799065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:44857 (local address 127.14.68.62:40659)
0927 10:53:57.799216 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:57.799222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:57.799308 (+ 86us) server_negotiation.cc:407] Connection header received
0927 10:53:57.799472 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:57.799475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:57.799525 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:57.799608 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:57.800093 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:57.800649 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:57.801587 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:57.801882 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:57.802003 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:57.804715 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.2:44857: BlockingRecv error: recv got EOF from 127.14.68.2:44857 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 10:53:57 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:57 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:57.999011 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:57.993854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.3:35429)
0927 10:53:57.994109 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:57.994123 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:57.994218 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:57.994440 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:57.994442 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:57.994623 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:57.994729 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:57.994734 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:57.995443 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:57.995446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:57.996087 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:57.996096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:57.996504 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:57.996506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:57.996562 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:57.997038 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:57.997048 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:57.998828 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":89,"threads_started":1}
W20260927 10:53:57.999125 23783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:57.993903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:35429 (local address 127.14.68.62:40659)
0927 10:53:57.994060 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:57.994064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:57.994198 (+ 134us) server_negotiation.cc:407] Connection header received
0927 10:53:57.994283 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:57.994285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:57.994332 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:57.994408 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:57.994823 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:57.995350 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:57.996187 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:57.996400 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:57.996490 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:57.999007 (+ 2517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.3:35429: BlockingRecv error: recv got EOF from 127.14.68.3:35429 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 27 10:53:58 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:58 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:58.628942 23786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:58.623180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.1:36789)
0927 10:53:58.623465 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:58.623479 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:58.623577 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:58.623903 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:58.623906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:58.624107 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:58.624200 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:58.624206 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:58.624952 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:58.624955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:58.625614 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:58.625622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:58.626043 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:58.626045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:58.626151 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:58.626643 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:58.626655 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:58.628781 (+ 2126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":78,"threads_started":1}
W20260927 10:53:58.629015 23787 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:58.623253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:36789 (local address 127.14.68.62:40659)
0927 10:53:58.623524 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:58.623527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:58.623541 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:53:58.623717 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:58.623719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:58.623769 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:58.623843 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:58.624312 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:58.624847 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:58.625728 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:58.625948 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:58.626025 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:58.628895 (+ 2870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.1:36789: BlockingRecv error: recv got EOF from 127.14.68.1:36789 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":82,"threads_started":1}
Sep 27 10:53:58 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:58 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:58.811465 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:58.805620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.2:55659)
0927 10:53:58.805883 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:58.805896 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:58.805994 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:58.806221 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:58.806223 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:58.806412 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:58.806504 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:58.806510 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:58.807266 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:58.807268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:58.807891 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:58.807900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:58.808386 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:58.808388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:58.808448 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:58.809020 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:58.809032 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:58.811271 (+ 2239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":90,"threads_started":1}
W20260927 10:53:58.811954 23787 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:58.805682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:55659 (local address 127.14.68.62:40659)
0927 10:53:58.805839 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:58.805843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:58.805992 (+ 149us) server_negotiation.cc:407] Connection header received
0927 10:53:58.806096 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:58.806098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:58.806145 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:58.806207 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:58.806606 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:58.807169 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:58.808018 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:58.808282 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:58.808373 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:58.811817 (+ 3444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.2:55659: BlockingRecv error: recv got EOF from 127.14.68.2:55659 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 10:53:59 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:59 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:59.005424 23789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:58.999950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.3:34941)
0927 10:53:59.000246 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:59.000260 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:59.000393 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:59.000673 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:59.000676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:59.000889 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:59.000984 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:59.000989 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:59.001757 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:59.001763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:59.002322 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:59.002329 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:59.002795 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:59.002799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:59.002887 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:59.003392 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:59.003403 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:59.005311 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":94,"threads_started":1}
W20260927 10:53:59.005596 23787 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:59.000002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:34941 (local address 127.14.68.62:40659)
0927 10:53:59.000171 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:59.000175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:59.000333 (+ 158us) server_negotiation.cc:407] Connection header received
0927 10:53:59.000492 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:59.000496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:59.000550 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:59.000636 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:59.001102 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:59.001642 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:59.002442 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:59.002672 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:59.002780 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:59.005441 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.3:34941: BlockingRecv error: recv got EOF from 127.14.68.3:34941 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 10:53:59 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:59 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 10:53:59.636053 23791 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:59.629811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:38245 (local address 127.14.68.62:40659)
0927 10:53:59.630094 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:59.630099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:59.630117 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:53:59.630254 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:59.630257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:59.630312 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:59.630407 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:59.630853 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:59.631406 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:59.632323 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:59.632563 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:59.632686 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:59.635951 (+ 3265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.1:38245: BlockingRecv error: recv got EOF from 127.14.68.1:38245 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":78,"threads_started":1}
I20260927 10:53:59.636052 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:59.629745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.1:38245)
0927 10:53:59.630036 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:59.630050 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:59.630136 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:59.630432 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:59.630436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:59.630639 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:59.630734 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:59.630740 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:59.631521 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:59.631525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:59.632196 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:59.632215 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:59.632681 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:59.632685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:59.632771 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:59.633357 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:59.633373 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:59.635830 (+ 2457us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":82,"threads_started":1}
Sep 27 10:53:59 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:53:59 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:53:59.817560 23792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:53:59.812203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.2:37363)
0927 10:53:59.812495 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 10:53:59.812507 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:53:59.812605 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:53:59.812826 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:53:59.812828 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:53:59.813010 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:53:59.813105 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:59.813110 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:59.813872 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:59.813875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:59.814467 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:53:59.814475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:53:59.814899 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:53:59.814902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:53:59.814964 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:59.815465 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:53:59.815476 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:53:59.817441 (+ 1965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":88,"threads_started":1}
W20260927 10:53:59.817781 23791 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:53:59.812299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:37363 (local address 127.14.68.62:40659)
0927 10:53:59.812437 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 10:53:59.812441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:53:59.812576 (+ 135us) server_negotiation.cc:407] Connection header received
0927 10:53:59.812684 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:53:59.812686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:53:59.812728 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:53:59.812786 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:53:59.813222 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:59.813750 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:59.814578 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:53:59.814800 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:53:59.814878 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:53:59.817633 (+ 2755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.2:37363: BlockingRecv error: recv got EOF from 127.14.68.2:37363 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 10:54:00 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:00 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:00.011864 23793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:00.006252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.3:34553)
0927 10:54:00.006582 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:00.006606 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:00.006738 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:00.007063 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:00.007066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:00.007310 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:00.007441 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:00.007466 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:00.008272 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:00.008275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:00.008902 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:00.008917 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:00.009309 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:00.009311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:00.009370 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:00.009848 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:00.009859 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:00.011679 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":101,"threads_started":1}
W20260927 10:54:00.011935 23791 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:00.006285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:34553 (local address 127.14.68.62:40659)
0927 10:54:00.006452 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:00.006456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:00.006699 (+ 243us) server_negotiation.cc:407] Connection header received
0927 10:54:00.006838 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:00.006841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:00.006899 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:00.007009 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:00.007573 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:00.008179 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:00.009006 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:00.009215 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:00.009292 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:00.011846 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.3:34553: BlockingRecv error: recv got EOF from 127.14.68.3:34553 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 10:54:00 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:00 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:00.642622 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:00.637024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.1:50141)
0927 10:54:00.637278 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:00.637296 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:00.637429 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:00.637781 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:00.637784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:00.638029 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:00.638199 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:00.638208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:00.639048 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:00.639051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:00.639631 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:00.639639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:00.640094 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:00.640097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:00.640184 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:00.640753 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:00.640765 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:00.642510 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":133,"threads_started":1}
W20260927 10:54:00.642823 23795 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:00.637201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:50141 (local address 127.14.68.62:40659)
0927 10:54:00.637452 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:00.637455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:00.637515 (+ 60us) server_negotiation.cc:407] Connection header received
0927 10:54:00.637559 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:00.637562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:00.637617 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:00.637720 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:00.638315 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:00.638951 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:00.639787 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:00.640001 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:00.640076 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:00.642638 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.1:50141: BlockingRecv error: recv got EOF from 127.14.68.1:50141 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":68,"threads_started":1}
Sep 27 10:54:00 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:00 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:00.824178 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:00.818480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.2:59879)
0927 10:54:00.818729 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:00.818747 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:00.818858 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:00.819149 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:00.819154 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:00.819387 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:00.819488 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:00.819494 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:00.820484 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:00.820488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:00.821064 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:00.821072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:00.821548 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:00.821551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:00.821638 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:00.822157 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:00.822185 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:00.824008 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":83,"threads_started":1}
W20260927 10:54:00.824236 23795 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:00.818528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:59879 (local address 127.14.68.62:40659)
0927 10:54:00.818658 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:00.818664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:00.818826 (+ 162us) server_negotiation.cc:407] Connection header received
0927 10:54:00.818970 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:00.818973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:00.819031 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:00.819103 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:00.819626 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:00.820370 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:00.821201 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:00.821437 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:00.821545 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:00.824122 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.2:59879: BlockingRecv error: recv got EOF from 127.14.68.2:59879 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 10:54:01 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:01 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:01.017987 23797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:01.012603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.3:36811)
0927 10:54:01.012903 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:01.012915 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:01.013053 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:01.013337 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:01.013340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:01.013559 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:01.013656 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:01.013663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:01.014397 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:01.014402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:01.014945 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:01.014952 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:01.015448 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:01.015452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:01.015537 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:01.016066 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:01.016078 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:01.017879 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":117,"threads_started":1}
W20260927 10:54:01.018280 23795 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:01.012697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:36811 (local address 127.14.68.62:40659)
0927 10:54:01.012843 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:01.012846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:01.012998 (+ 152us) server_negotiation.cc:407] Connection header received
0927 10:54:01.013154 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:01.013158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:01.013218 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:01.013309 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:01.013783 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:01.014290 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:01.015058 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:01.015332 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:01.015444 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:01.017996 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.3:36811: BlockingRecv error: recv got EOF from 127.14.68.3:36811 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 10:54:01 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:01 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:01.653926 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:01.643317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.1:36329)
0927 10:54:01.643606 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:01.643624 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:01.643792 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:01.644021 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:01.644024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:01.644199 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:01.644289 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:01.644293 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:01.645001 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:01.645004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:01.645545 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:01.645552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:01.645949 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:01.645952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:01.646014 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:01.651465 (+ 5451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:01.651477 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:01.653815 (+ 2338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
W20260927 10:54:01.654147 23799 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:01.643454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:36329 (local address 127.14.68.62:40659)
0927 10:54:01.643743 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:01.643747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:01.643759 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:54:01.643871 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:01.643873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:01.643913 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:01.643985 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:01.644381 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:01.644894 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:01.645638 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:01.645859 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:01.645930 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:01.653956 (+ 8026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.1:36329: BlockingRecv error: recv got EOF from 127.14.68.1:36329 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":102,"threads_started":1}
Sep 27 10:54:01 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:01 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:01.830487 23800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:01.824903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.2:42505)
0927 10:54:01.825172 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:01.825185 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:01.825285 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:01.825520 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:01.825522 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:01.825720 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:01.825817 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:01.825823 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:01.826560 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:01.826563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:01.827288 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:01.827299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:01.827951 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:01.827954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:01.828063 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:01.828613 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:01.828624 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:01.830372 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":102,"threads_started":1}
W20260927 10:54:01.830623 23799 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:01.824990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:42505 (local address 127.14.68.62:40659)
0927 10:54:01.825128 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:01.825132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:01.825253 (+ 121us) server_negotiation.cc:407] Connection header received
0927 10:54:01.825363 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:01.825365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:01.825410 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:01.825480 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:01.825916 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:01.826452 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:01.827402 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:01.827755 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:01.827847 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:01.830503 (+ 2656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.2:42505: BlockingRecv error: recv got EOF from 127.14.68.2:42505 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 10:54:02 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:02 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:02.024693 23801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:02.018760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.3:47149)
0927 10:54:02.019045 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:02.019058 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:02.019221 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:02.019523 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:02.019526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:02.019779 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:02.019871 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:02.019877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:02.020622 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:02.020628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:02.021200 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:02.021208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:02.021656 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:02.021660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:02.021744 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:02.022314 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:02.022327 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:02.024516 (+ 2189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":88,"threads_started":1}
W20260927 10:54:02.024835 23799 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:02.018812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:47149 (local address 127.14.68.62:40659)
0927 10:54:02.018972 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:02.018977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:02.019137 (+ 160us) server_negotiation.cc:407] Connection header received
0927 10:54:02.019334 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:02.019338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:02.019393 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:02.019502 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:02.019988 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:02.020513 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:02.021321 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:02.021543 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:02.021653 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:02.024690 (+ 3037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.3:47149: BlockingRecv error: recv got EOF from 127.14.68.3:47149 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 10:54:02 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:02 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:02.660465 23802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:02.654651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.1:53139)
0927 10:54:02.654961 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:02.654976 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:02.655077 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:02.655345 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:02.655348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:02.655535 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:02.655635 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:02.655641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:02.656463 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:02.656466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:02.657024 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:02.657032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:02.657512 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:02.657515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:02.657595 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:02.658333 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:02.658346 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:02.660356 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":91,"threads_started":1}
W20260927 10:54:02.660678 23803 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:02.654752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:53139 (local address 127.14.68.62:40659)
0927 10:54:02.654987 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:02.654990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:02.655030 (+ 40us) server_negotiation.cc:407] Connection header received
0927 10:54:02.655146 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:02.655147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:02.655194 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:02.655268 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:02.655811 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:02.656366 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:02.657179 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:02.657392 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:02.657492 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:02.660498 (+ 3006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.1:53139: BlockingRecv error: recv got EOF from 127.14.68.1:53139 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":58,"threads_started":1}
Sep 27 10:54:02 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:02 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:02.836695 23804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:02.831293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.2:41155)
0927 10:54:02.831621 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:02.831643 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:02.831814 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:02.832086 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:02.832088 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:02.832288 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:02.832383 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:02.832388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:02.833101 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:02.833104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:02.833663 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:02.833670 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:02.834075 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:02.834078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:02.834139 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:02.834648 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:02.834659 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:02.836544 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":96,"threads_started":1}
W20260927 10:54:02.836795 23803 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:02.831414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:41155 (local address 127.14.68.62:40659)
0927 10:54:02.831592 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:02.831597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:02.831771 (+ 174us) server_negotiation.cc:407] Connection header received
0927 10:54:02.831889 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:02.831891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:02.831938 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:02.832010 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:02.832491 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:02.833009 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:02.833765 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:02.833980 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:02.834056 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:02.836659 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.2:41155: BlockingRecv error: recv got EOF from 127.14.68.2:41155 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 10:54:03 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:03 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:03.031236 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:03.025462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.3:34511)
0927 10:54:03.025803 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:03.025819 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:03.025929 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:03.026237 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:03.026240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:03.026451 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:03.026541 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:03.026547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:03.027295 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:03.027301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:03.027975 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:03.027985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:03.028565 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:03.028572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:03.028680 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:03.029260 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:03.029272 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:03.031030 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":115,"threads_started":1}
W20260927 10:54:03.031275 23803 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:03.025644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:34511 (local address 127.14.68.62:40659)
0927 10:54:03.025803 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:03.025808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:03.025881 (+ 73us) server_negotiation.cc:407] Connection header received
0927 10:54:03.026040 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:03.026045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:03.026107 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:03.026211 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:03.026671 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:03.027186 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:03.028119 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:03.028455 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:03.028539 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:03.031175 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.3:34511: BlockingRecv error: recv got EOF from 127.14.68.3:34511 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 10:54:03 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:03 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:03.666427 23806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:03.661264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.1:47105)
0927 10:54:03.661532 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:03.661545 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:03.661640 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:03.661887 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:03.661889 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:03.662065 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:03.662167 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:03.662171 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:03.662869 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:03.662871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:03.663452 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:03.663459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:03.663879 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:03.663881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:03.663943 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:03.664434 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:03.664445 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:03.666314 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":68,"threads_started":1}
W20260927 10:54:03.666615 23807 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:03.661305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:47105 (local address 127.14.68.62:40659)
0927 10:54:03.661573 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:03.661577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:03.661590 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:54:03.661728 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:03.661731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:03.661775 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:03.661848 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:03.662265 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:03.662783 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:03.663548 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:03.663786 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:03.663858 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:03.666444 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.1:47105: BlockingRecv error: recv got EOF from 127.14.68.1:47105 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":98,"threads_started":1}
Sep 27 10:54:03 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:03 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:03.842932 23808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:03.837505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.2:56545)
0927 10:54:03.837780 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:03.837794 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:03.837890 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:03.838120 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:03.838122 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:03.838313 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:03.838406 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:03.838411 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:03.839134 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:03.839136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:03.839733 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:03.839742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:03.840149 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:03.840151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:03.840225 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:03.840684 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:03.840696 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:03.842817 (+ 2121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
W20260927 10:54:03.843066 23807 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:03.837551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:56545 (local address 127.14.68.62:40659)
0927 10:54:03.837698 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:03.837703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:03.837868 (+ 165us) server_negotiation.cc:407] Connection header received
0927 10:54:03.837970 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:03.837972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:03.838016 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:03.838083 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:03.838500 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:03.839026 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:03.839844 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:03.840056 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:03.840132 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:03.842950 (+ 2818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.2:56545: BlockingRecv error: recv got EOF from 127.14.68.2:56545 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 10:54:04 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:04 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:04.037482 23809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:04.031991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.3:50799)
0927 10:54:04.032278 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:04.032291 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:04.032400 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:04.032611 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:04.032614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:04.032810 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:04.032901 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:04.032906 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:04.033624 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:04.033627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:04.034211 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:04.034217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:04.034653 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:04.034656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:04.034763 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:04.035404 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:04.035418 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:04.037369 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":79,"threads_started":1}
W20260927 10:54:04.037688 23807 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:04.032066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:50799 (local address 127.14.68.62:40659)
0927 10:54:04.032193 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:04.032196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:04.032371 (+ 175us) server_negotiation.cc:407] Connection header received
0927 10:54:04.032464 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:04.032466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:04.032509 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:04.032573 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:04.033007 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:04.033532 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:04.034318 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:04.034520 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:04.034642 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:04.037498 (+ 2856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.3:50799: BlockingRecv error: recv got EOF from 127.14.68.3:50799 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 10:54:04 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:04 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:04.672300 23810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:04.667254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.1:33219)
0927 10:54:04.667562 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:04.667576 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:04.667670 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:04.667945 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:04.667947 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:04.668126 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:04.668226 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:04.668230 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:04.668925 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:04.668927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:04.669488 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:04.669495 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:04.669889 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:04.669891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:04.669952 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:04.670427 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:04.670438 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:04.672187 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":83,"threads_started":1}
W20260927 10:54:04.672459 23811 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:04.667333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:33219 (local address 127.14.68.62:40659)
0927 10:54:04.667612 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:04.667615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:04.667628 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:54:04.667776 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:04.667778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:04.667820 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:04.667905 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:04.668317 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:04.668815 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:04.669584 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:04.669797 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:04.669868 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:04.672323 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.1:33219: BlockingRecv error: recv got EOF from 127.14.68.1:33219 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":113,"threads_started":1}
Sep 27 10:54:04 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:04 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:04.849417 23812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:04.843665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.2:43039)
0927 10:54:04.844022 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:04.844037 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:04.844183 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:04.844517 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:04.844520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:04.844749 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:04.844857 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:04.844863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:04.845626 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:04.845631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:04.846218 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:04.846226 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:04.846708 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:04.846712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:04.846799 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:04.847374 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:04.847387 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:04.849301 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":102,"threads_started":1}
W20260927 10:54:04.849546 23811 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:04.843804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:43039 (local address 127.14.68.62:40659)
0927 10:54:04.843977 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:04.843982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:04.844124 (+ 142us) server_negotiation.cc:407] Connection header received
0927 10:54:04.844301 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:04.844305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:04.844394 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:04.844488 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:04.844981 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:04.845514 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:04.846341 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:04.846586 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:04.846696 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:04.849432 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.2:43039: BlockingRecv error: recv got EOF from 127.14.68.2:43039 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 10:54:05 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:05 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:05.043509 23813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:05.038236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.3:42633)
0927 10:54:05.038550 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:05.038564 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:05.038677 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:05.038910 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:05.038912 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:05.039087 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:05.039177 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:05.039182 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:05.039877 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:05.039879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:05.040439 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:05.040446 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:05.040823 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:05.040825 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:05.040884 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:05.041374 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:05.041385 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:05.043348 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":118,"threads_started":1}
W20260927 10:54:05.043568 23811 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:05.038306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:42633 (local address 127.14.68.62:40659)
0927 10:54:05.038451 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:05.038456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:05.038642 (+ 186us) server_negotiation.cc:407] Connection header received
0927 10:54:05.038760 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:05.038762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:05.038809 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:05.038867 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:05.039273 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:05.039797 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:05.040532 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:05.040734 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:05.040804 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:05.043460 (+ 2656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.3:42633: BlockingRecv error: recv got EOF from 127.14.68.3:42633 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 10:54:05 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:05 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:05.678848 23814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:05.673147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.1:60671)
0927 10:54:05.673480 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:05.673494 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:05.673591 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:05.673881 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:05.673883 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:05.674079 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:05.674167 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:05.674174 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:05.675208 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:05.675214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:05.675914 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:05.675924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:05.676355 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:05.676358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:05.676417 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:05.676894 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:05.676906 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:05.678712 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":105,"threads_started":1}
W20260927 10:54:05.678987 23815 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:05.673275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:60671 (local address 127.14.68.62:40659)
0927 10:54:05.673510 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:05.673514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:05.673545 (+ 31us) server_negotiation.cc:407] Connection header received
0927 10:54:05.673734 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:05.673737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:05.673786 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:05.673871 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:05.674336 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:05.675065 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:05.676015 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:05.676261 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:05.676341 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:05.678850 (+ 2509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.1:60671: BlockingRecv error: recv got EOF from 127.14.68.1:60671 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":61,"threads_started":1}
Sep 27 10:54:05 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:05 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:05.855858 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:05.850377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.2:46049)
0927 10:54:05.850674 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:05.850688 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:05.850812 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:05.851102 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:05.851104 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:05.851306 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:05.851404 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:05.851409 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:05.852173 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:05.852176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:05.852733 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:05.852740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:05.853120 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:05.853122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:05.853205 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:05.853751 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:05.853762 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:05.855633 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":84,"threads_started":1}
W20260927 10:54:05.856138 23815 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:05.850532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:46049 (local address 127.14.68.62:40659)
0927 10:54:05.850703 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:05.850707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:05.850760 (+ 53us) server_negotiation.cc:407] Connection header received
0927 10:54:05.850936 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:05.850939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:05.850995 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:05.851061 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:05.851508 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:05.852052 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:05.852825 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:05.853030 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:05.853103 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:05.855917 (+ 2814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.2:46049: BlockingRecv error: recv got EOF from 127.14.68.2:46049 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 10:54:06 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:06 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:06.049742 23817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:06.044318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.3:53409)
0927 10:54:06.044605 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:06.044618 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:06.044759 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:06.045030 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:06.045033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:06.045263 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:06.045352 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:06.045358 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:06.046136 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:06.046142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:06.046692 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:06.046699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:06.047145 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:06.047149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:06.047245 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:06.047894 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:06.047905 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:06.049621 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
W20260927 10:54:06.049865 23815 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:06.044382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:53409 (local address 127.14.68.62:40659)
0927 10:54:06.044552 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:06.044556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:06.044700 (+ 144us) server_negotiation.cc:407] Connection header received
0927 10:54:06.044859 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:06.044862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:06.044917 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:06.045005 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:06.045472 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:06.046000 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:06.046812 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:06.047035 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:06.047144 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:06.049748 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.3:53409: BlockingRecv error: recv got EOF from 127.14.68.3:53409 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 10:54:06 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:06 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:06.691136 23818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:06.679744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.1:60121)
0927 10:54:06.680063 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:06.680079 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:06.680202 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:06.680621 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:06.680625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:06.680884 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:06.681000 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:06.681007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:06.682078 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:06.682081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:06.682741 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:06.682751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:06.683448 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:06.683454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:06.683578 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:06.688807 (+ 5229us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:06.688822 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:06.690978 (+ 2156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":85,"threads_started":1}
W20260927 10:54:06.691274 23819 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:06.679774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:60121 (local address 127.14.68.62:40659)
0927 10:54:06.680078 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:06.680083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:06.680168 (+ 85us) server_negotiation.cc:407] Connection header received
0927 10:54:06.680332 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:06.680372 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:06.680453 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:06.680545 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:06.681138 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:06.681945 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:06.682863 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:06.683312 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:06.683431 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:06.691128 (+ 7697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.1:60121: BlockingRecv error: recv got EOF from 127.14.68.1:60121 (error 108)
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":55,"threads_started":1}
Sep 27 10:54:06 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:06 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:06.862118 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:06.856656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.2:38869)
0927 10:54:06.856934 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:06.856947 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:06.857082 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:06.857351 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:06.857354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:06.857608 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:06.857715 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:06.857721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:06.858500 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:06.858505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:06.859094 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:06.859101 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:06.859560 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:06.859563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:06.859648 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:06.860138 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:06.860149 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:06.862005 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
W20260927 10:54:06.862313 23819 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:06.856712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:38869 (local address 127.14.68.62:40659)
0927 10:54:06.856878 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:06.856883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:06.857018 (+ 135us) server_negotiation.cc:407] Connection header received
0927 10:54:06.857179 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:06.857182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:06.857238 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:06.857332 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:06.857827 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:06.858392 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:06.859221 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:06.859449 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:06.859556 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:06.862173 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.2:38869: BlockingRecv error: recv got EOF from 127.14.68.2:38869 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 10:54:07 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:07 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.3@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:07.056134 23821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:07.050519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.3:39855)
0927 10:54:07.050813 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:07.050827 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:07.050929 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:07.051160 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:07.051163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:07.051350 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:07.051454 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:07.051459 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:07.052232 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:07.052235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:07.052912 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:07.052922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:07.053339 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:07.053341 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:07.053414 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:07.053921 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:07.053934 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:07.056005 (+ 2071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
W20260927 10:54:07.056284 23819 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:07.050646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:39855 (local address 127.14.68.62:40659)
0927 10:54:07.050819 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:07.050822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:07.050898 (+ 76us) server_negotiation.cc:407] Connection header received
0927 10:54:07.051009 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:07.051011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:07.051056 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:07.051120 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:07.051550 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:07.052114 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:07.053023 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:07.053234 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:07.053306 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:07.056138 (+ 2832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.3:39855: BlockingRecv error: recv got EOF from 127.14.68.3:39855 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 10:54:07 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:07 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.1@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:07.697774 23822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:07.692280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.1:60471)
0927 10:54:07.692563 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:07.692580 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:07.692677 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:07.692989 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:07.692993 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:07.693222 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:07.693317 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:07.693322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:07.694119 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:07.694121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:07.694679 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:07.694686 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:07.695087 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:07.695089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:07.695151 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:07.695654 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:07.695666 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:07.697660 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":117,"threads_started":1}
W20260927 10:54:07.697949 23823 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:07.692369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:60471 (local address 127.14.68.62:40659)
0927 10:54:07.692589 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:07.692592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:07.692646 (+ 54us) server_negotiation.cc:407] Connection header received
0927 10:54:07.692798 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:07.692800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:07.692845 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:07.692921 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:07.693427 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:07.694032 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:07.694777 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:07.694994 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:07.695069 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:07.697809 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.1:60471: BlockingRecv error: recv got EOF from 127.14.68.1:60471 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":83,"threads_started":1}
Sep 27 10:54:07 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:54:07 dist-test-slave-tz51 krb5kdc[22764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.68.2@KRBTEST.COM for oryx/127.14.68.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:54:07.868443 23824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:07.863117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40659 (local address 127.14.68.2:33063)
0927 10:54:07.863465 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:07.863480 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:07.863597 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:07.863908 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:07.863911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:07.864130 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:07.864229 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:07.864235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:07.865015 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:07.865018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:07.865606 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:07.865615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:07.866009 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:07.866012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:07.866108 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:07.866576 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:07.866590 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:07.868333 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.68.62:40659: Server oryx/127.14.68.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":88,"threads_started":1}
W20260927 10:54:07.868580 23823 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:07.863171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:33063 (local address 127.14.68.62:40659)
0927 10:54:07.863321 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:07.863327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:07.863582 (+ 255us) server_negotiation.cc:407] Connection header received
0927 10:54:07.863688 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:07.863732 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:07.863785 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:07.863872 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:07.864386 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:07.864908 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:07.865709 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:07.865920 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:07.865993 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:07.868462 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.68.2:33063: BlockingRecv error: recv got EOF from 127.14.68.2:33063 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:54:07.964371 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 23347
I20260927 10:54:07.969857 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 23498
I20260927 10:54:07.975123 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 23631
I20260927 10:54:07.980347 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 23277
2026-09-27T10:54:07Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17944 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T10:54:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:08Z Disabled control of system clock
I20260927 10:54:08.010524 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:33995
--webserver_interface=127.14.68.62
--webserver_port=0
--builtin_ntp_servers=127.14.68.20:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:33995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 10:54:08.116608 23828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:08.116853 23828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:08.116885 23828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:08.120751 23828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:08.120800 23828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:08.120815 23828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:08.120831 23828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:08.120846 23828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:08.125586 23828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:33995
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:33995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:54:08.126685 23828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:08.128059 23828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:54:08.136703 23828 server_base.cc:1034] running on GCE node
W20260927 10:54:08.136668 23836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:08.136648 23834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:08.136660 23833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:08.137507 23828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:08.138517 23828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:08.139678 23828 hybrid_clock.cc:648] HybridClock initialized: now 1790506448139663 us; error 27 us; skew 500 ppm
I20260927 10:54:08.141546 23828 webserver.cc:460] Webserver started at http://127.14.68.62:46497/ using document root <none> and password file <none>
I20260927 10:54:08.142117 23828 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:08.142207 23828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:08.142427 23828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:08.144515 23828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "04f27984eda845d0b889a258e7d6180b"
format_stamp: "Formatted at 2026-09-27 10:54:08 on dist-test-slave-tz51"
server_key: "3bd3579629ed10701c8d5b5e13307b24"
server_key_iv: "fe55b6781a2dbd0739e59c48d809f821"
server_key_version: "encryptionkey@0"
I20260927 10:54:08.145079 23828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "04f27984eda845d0b889a258e7d6180b"
format_stamp: "Formatted at 2026-09-27 10:54:08 on dist-test-slave-tz51"
server_key: "3bd3579629ed10701c8d5b5e13307b24"
server_key_iv: "fe55b6781a2dbd0739e59c48d809f821"
server_key_version: "encryptionkey@0"
I20260927 10:54:08.148722 23828 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:54:08.151064 23842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:08.152354 23828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:08.152488 23828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "04f27984eda845d0b889a258e7d6180b"
format_stamp: "Formatted at 2026-09-27 10:54:08 on dist-test-slave-tz51"
server_key: "3bd3579629ed10701c8d5b5e13307b24"
server_key_iv: "fe55b6781a2dbd0739e59c48d809f821"
server_key_version: "encryptionkey@0"
I20260927 10:54:08.152598 23828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:08.166102 23828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:08.166719 23828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:08.166908 23828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:08.173828 23828 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:33995
I20260927 10:54:08.173835 23894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:33995 every 8 connection(s)
I20260927 10:54:08.174757 23828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:08.175845 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 23828
I20260927 10:54:08.175968 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:54:08.176271 14608 external_mini_cluster.cc:1428] Setting key 11f97dbc03c73a5a36a77174391a510e
I20260927 10:54:08.178606 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:08.183915 23895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04f27984eda845d0b889a258e7d6180b: Bootstrap starting.
I20260927 10:54:08.185240 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:08.177666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46356 (local address 127.14.68.62:33995)
0927 10:54:08.178312 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:08.178328 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:08.178361 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:54:08.179070 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:08.179087 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:08.179466 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:08.179834 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:08.180252 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:08.181197 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:08.182083 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:08.182545 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:08.182715 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:08.183578 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:08.183583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:08.183594 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:08.184049 (+ 455us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:08.184116 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:08.184144 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:08.184247 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:08.184358 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 10:54:08.184598 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":158,"threads_started":1}
I20260927 10:54:08.186096 23895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04f27984eda845d0b889a258e7d6180b: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:08.186842 23895 log.cc:824] T 00000000000000000000000000000000 P 04f27984eda845d0b889a258e7d6180b: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:08.188741 23895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04f27984eda845d0b889a258e7d6180b: No bootstrap required, opened a new log
I20260927 10:54:08.191962 23895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04f27984eda845d0b889a258e7d6180b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f27984eda845d0b889a258e7d6180b" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33995 } }
I20260927 10:54:08.192199 23895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04f27984eda845d0b889a258e7d6180b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:08.192276 23895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04f27984eda845d0b889a258e7d6180b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f27984eda845d0b889a258e7d6180b, State: Initialized, Role: FOLLOWER
I20260927 10:54:08.192706 23895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04f27984eda845d0b889a258e7d6180b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f27984eda845d0b889a258e7d6180b" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33995 } }
I20260927 10:54:08.192888 23895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04f27984eda845d0b889a258e7d6180b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:08.192970 23895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04f27984eda845d0b889a258e7d6180b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:08.193091 23895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04f27984eda845d0b889a258e7d6180b [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:08.193881 23895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04f27984eda845d0b889a258e7d6180b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f27984eda845d0b889a258e7d6180b" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33995 } }
I20260927 10:54:08.194239 23895 leader_election.cc:302] T 00000000000000000000000000000000 P 04f27984eda845d0b889a258e7d6180b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04f27984eda845d0b889a258e7d6180b; no voters:
I20260927 10:54:08.194545 23895 leader_election.cc:288] T 00000000000000000000000000000000 P 04f27984eda845d0b889a258e7d6180b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:08.194689 23900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04f27984eda845d0b889a258e7d6180b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:08.194971 23900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04f27984eda845d0b889a258e7d6180b [term 1 LEADER]: Becoming Leader. State: Replica: 04f27984eda845d0b889a258e7d6180b, State: Running, Role: LEADER
I20260927 10:54:08.195281 23900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04f27984eda845d0b889a258e7d6180b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f27984eda845d0b889a258e7d6180b" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33995 } }
I20260927 10:54:08.195855 23895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04f27984eda845d0b889a258e7d6180b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:08.197105 23901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04f27984eda845d0b889a258e7d6180b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04f27984eda845d0b889a258e7d6180b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f27984eda845d0b889a258e7d6180b" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33995 } } }
I20260927 10:54:08.197116 23902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04f27984eda845d0b889a258e7d6180b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04f27984eda845d0b889a258e7d6180b. Latest consensus state: current_term: 1 leader_uuid: "04f27984eda845d0b889a258e7d6180b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f27984eda845d0b889a258e7d6180b" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33995 } } }
I20260927 10:54:08.197285 23901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04f27984eda845d0b889a258e7d6180b [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:08.197316 23902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04f27984eda845d0b889a258e7d6180b [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:08.197676 23910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:08.200233 23910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:08.205539 23910 catalog_manager.cc:1347] Generated new cluster ID: 58a1f72cda6e49d6b84f774a1120d23f
I20260927 10:54:08.205601 23910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:08.224517 23910 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:08.225603 23910 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:08.236872 23910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04f27984eda845d0b889a258e7d6180b: Generated new TSK 0
I20260927 10:54:08.237584 23910 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:54:08.250497 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:33995
--builtin_ntp_servers=127.14.68.20:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 10:54:08.359853 23919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:08.360093 23919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:08.360128 23919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:08.364159 23919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:08.364208 23919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:08.364295 23919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:54:08.368929 23919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:33995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:54:08.369933 23919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:08.371292 23919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:08.380559 23925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:08.380597 23924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:08.380788 23919 server_base.cc:1034] running on GCE node
W20260927 10:54:08.380959 23927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:08.381325 23919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:08.381894 23919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:08.383046 23919 hybrid_clock.cc:648] HybridClock initialized: now 1790506448383032 us; error 25 us; skew 500 ppm
I20260927 10:54:08.384804 23919 webserver.cc:460] Webserver started at http://127.14.68.1:33719/ using document root <none> and password file <none>
I20260927 10:54:08.385350 23919 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:08.385411 23919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:08.385641 23919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:08.387619 23919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "74b8a68a07304d1889a522c255fa13a1"
format_stamp: "Formatted at 2026-09-27 10:54:08 on dist-test-slave-tz51"
server_key: "f7ea3150b10c63c10ef2728d5bf2b896"
server_key_iv: "f0cdc03e4543c8065eacc0b05136cede"
server_key_version: "encryptionkey@0"
I20260927 10:54:08.388224 23919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "74b8a68a07304d1889a522c255fa13a1"
format_stamp: "Formatted at 2026-09-27 10:54:08 on dist-test-slave-tz51"
server_key: "f7ea3150b10c63c10ef2728d5bf2b896"
server_key_iv: "f0cdc03e4543c8065eacc0b05136cede"
server_key_version: "encryptionkey@0"
I20260927 10:54:08.391629 23919 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:54:08.394022 23933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:08.395187 23919 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:08.395310 23919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "74b8a68a07304d1889a522c255fa13a1"
format_stamp: "Formatted at 2026-09-27 10:54:08 on dist-test-slave-tz51"
server_key: "f7ea3150b10c63c10ef2728d5bf2b896"
server_key_iv: "f0cdc03e4543c8065eacc0b05136cede"
server_key_version: "encryptionkey@0"
I20260927 10:54:08.395417 23919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:08.408664 23919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:08.409318 23919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:08.409503 23919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:08.410082 23919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:08.411208 23919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:08.411257 23919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:08.411320 23919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:08.411351 23919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:08.420055 23919 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:43767
I20260927 10:54:08.420082 24046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:43767 every 8 connection(s)
I20260927 10:54:08.420908 23919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:54:08.426927 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 23919
I20260927 10:54:08.427022 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:54:08.427335 14608 external_mini_cluster.cc:1428] Setting key ddc01b7a9b2649eb24d858a771d892bc
I20260927 10:54:08.429165 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:33995
--builtin_ntp_servers=127.14.68.20:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 10:54:08.430085 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:08.422683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:46277 (local address 127.14.68.62:33995)
0927 10:54:08.422816 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:08.422820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:08.423857 (+ 1037us) server_negotiation.cc:407] Connection header received
0927 10:54:08.425195 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:08.425198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:08.425243 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:08.425316 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:08.426115 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:08.426630 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:08.428007 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:08.428266 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:08.428386 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:08.429580 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:08.429584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:08.429586 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:08.429600 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:08.429625 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:08.429633 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:08.429716 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:08.429873 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 10:54:08.429949 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:54:08.430927 24049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:08.423059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:33995 (local address 127.14.68.1:46277)
0927 10:54:08.423668 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:08.423756 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:08.425015 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:08.425441 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:08.425447 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:08.425527 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:08.425981 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:08.425993 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:08.426768 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:08.426770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:08.427911 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:08.427920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:08.428420 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:08.428424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:08.428623 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:08.429280 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:54:08.429286 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:08.429343 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:54:08.429422 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:08.429718 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:08.429730 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:08.429737 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 10:54:08.429872 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 10:54:08.430123 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":106,"threads_started":1}
I20260927 10:54:08.432094 24047 heartbeater.cc:347] Connected to a master server at 127.14.68.62:33995
I20260927 10:54:08.432325 24047 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:08.432909 24047 heartbeater.cc:511] Master 127.14.68.62:33995 requested a full tablet report, sending...
I20260927 10:54:08.434423 23859 ts_manager.cc:194] Registered new tserver with Master: 74b8a68a07304d1889a522c255fa13a1 (127.14.68.1:43767)
I20260927 10:54:08.435539 23859 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.68.1:46277
W20260927 10:54:08.539386 24050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:08.539626 24050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:08.539669 24050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:08.544116 24050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:08.544198 24050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:08.544286 24050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:54:08.548897 24050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:33995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:54:08.549894 24050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:08.551189 24050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:08.561254 24056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:08.561359 24050 server_base.cc:1034] running on GCE node
W20260927 10:54:08.561287 24058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:08.561288 24055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:08.561837 24050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:08.562428 24050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:08.563612 24050 hybrid_clock.cc:648] HybridClock initialized: now 1790506448563599 us; error 30 us; skew 500 ppm
I20260927 10:54:08.566048 24050 webserver.cc:460] Webserver started at http://127.14.68.2:39089/ using document root <none> and password file <none>
I20260927 10:54:08.566658 24050 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:08.566758 24050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:08.566987 24050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:08.569051 24050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "fe0fb3a675fb4ec7aac1f7f7a9461cc0"
format_stamp: "Formatted at 2026-09-27 10:54:08 on dist-test-slave-tz51"
server_key: "6d69d0b8ffd9ce88f416b3f90dc30818"
server_key_iv: "ac193418c82b639d7958ea0f898068e1"
server_key_version: "encryptionkey@0"
I20260927 10:54:08.569594 24050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "fe0fb3a675fb4ec7aac1f7f7a9461cc0"
format_stamp: "Formatted at 2026-09-27 10:54:08 on dist-test-slave-tz51"
server_key: "6d69d0b8ffd9ce88f416b3f90dc30818"
server_key_iv: "ac193418c82b639d7958ea0f898068e1"
server_key_version: "encryptionkey@0"
I20260927 10:54:08.573256 24050 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 10:54:08.575810 24064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:08.577059 24050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:54:08.577200 24050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "fe0fb3a675fb4ec7aac1f7f7a9461cc0"
format_stamp: "Formatted at 2026-09-27 10:54:08 on dist-test-slave-tz51"
server_key: "6d69d0b8ffd9ce88f416b3f90dc30818"
server_key_iv: "ac193418c82b639d7958ea0f898068e1"
server_key_version: "encryptionkey@0"
I20260927 10:54:08.577319 24050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:08.589915 24050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:08.590595 24050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:08.590796 24050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:08.591363 24050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:08.592368 24050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:08.592439 24050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:08.592507 24050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:08.592555 24050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:08.601603 24050 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:39123
I20260927 10:54:08.601631 24177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:39123 every 8 connection(s)
I20260927 10:54:08.602496 24050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:54:08.605171 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 24050
I20260927 10:54:08.605284 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:54:08.605593 14608 external_mini_cluster.cc:1428] Setting key 4743fa92d5f3e4a2de3c99d327e92232
I20260927 10:54:08.607949 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:33995
--builtin_ntp_servers=127.14.68.20:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 10:54:08.612619 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:08.604371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:37539 (local address 127.14.68.62:33995)
0927 10:54:08.604510 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:08.604513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:08.605788 (+ 1275us) server_negotiation.cc:407] Connection header received
0927 10:54:08.607465 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:08.607467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:08.607511 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:08.607582 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:08.608816 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:08.609343 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:08.610523 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:08.610759 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:08.610872 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:08.612077 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:08.612087 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:08.612090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:08.612102 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:08.612159 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:08.612168 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:08.612250 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:08.612411 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 10:54:08.612477 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:54:08.613456 24180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:08.604769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:33995 (local address 127.14.68.2:37539)
0927 10:54:08.605601 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:08.605662 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:08.607236 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:08.607816 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:08.607827 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:08.607941 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:08.608660 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:08.608683 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:08.609496 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:08.609499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:08.610393 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:08.610403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:08.610910 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:08.610914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:08.611129 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:08.611821 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:54:08.611823 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:08.611863 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:54:08.611926 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:08.612252 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:08.612264 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:08.612272 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 10:54:08.612400 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 10:54:08.612704 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":116,"threads_started":1}
I20260927 10:54:08.614645 24178 heartbeater.cc:347] Connected to a master server at 127.14.68.62:33995
I20260927 10:54:08.614916 24178 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:08.615429 24178 heartbeater.cc:511] Master 127.14.68.62:33995 requested a full tablet report, sending...
I20260927 10:54:08.616545 23859 ts_manager.cc:194] Registered new tserver with Master: fe0fb3a675fb4ec7aac1f7f7a9461cc0 (127.14.68.2:39123)
I20260927 10:54:08.617434 23859 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.68.2:37539
W20260927 10:54:08.718580 24181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:08.718854 24181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:08.718891 24181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:08.722862 24181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:08.722919 24181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:08.722971 24181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:54:08.727622 24181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:33995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:54:08.728636 24181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:08.729925 24181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:08.739899 24189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:08.739886 24186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:08.739904 24187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:08.740331 24181 server_base.cc:1034] running on GCE node
I20260927 10:54:08.740705 24181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:08.741322 24181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:08.742504 24181 hybrid_clock.cc:648] HybridClock initialized: now 1790506448742489 us; error 31 us; skew 500 ppm
I20260927 10:54:08.744457 24181 webserver.cc:460] Webserver started at http://127.14.68.3:33809/ using document root <none> and password file <none>
I20260927 10:54:08.745023 24181 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:08.745113 24181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:08.745360 24181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:08.747354 24181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "4919661b97a247a1bbc8415a7de883bd"
format_stamp: "Formatted at 2026-09-27 10:54:08 on dist-test-slave-tz51"
server_key: "45c0ac83d11de53153be4191f8fab5b8"
server_key_iv: "dd59051cbd1253c9370447c6cb1e0008"
server_key_version: "encryptionkey@0"
I20260927 10:54:08.747946 24181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "4919661b97a247a1bbc8415a7de883bd"
format_stamp: "Formatted at 2026-09-27 10:54:08 on dist-test-slave-tz51"
server_key: "45c0ac83d11de53153be4191f8fab5b8"
server_key_iv: "dd59051cbd1253c9370447c6cb1e0008"
server_key_version: "encryptionkey@0"
I20260927 10:54:08.751505 24181 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:54:08.753984 24195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:08.755106 24181 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:08.755256 24181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "4919661b97a247a1bbc8415a7de883bd"
format_stamp: "Formatted at 2026-09-27 10:54:08 on dist-test-slave-tz51"
server_key: "45c0ac83d11de53153be4191f8fab5b8"
server_key_iv: "dd59051cbd1253c9370447c6cb1e0008"
server_key_version: "encryptionkey@0"
I20260927 10:54:08.755370 24181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:08.769580 24181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:08.770228 24181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:08.770445 24181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:08.770987 24181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:08.771932 24181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:08.771999 24181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:08.772068 24181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:08.772115 24181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:08.781518 24181 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:36143
I20260927 10:54:08.781556 24308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:36143 every 8 connection(s)
I20260927 10:54:08.782384 24181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:54:08.784415 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 24181
I20260927 10:54:08.784515 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:54:08.784791 14608 external_mini_cluster.cc:1428] Setting key 6fea86a9fb37cf1b79946bbbd2d09f92
I20260927 10:54:08.791932 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:08.784265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:39363 (local address 127.14.68.62:33995)
0927 10:54:08.784435 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:08.784441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:08.785685 (+ 1244us) server_negotiation.cc:407] Connection header received
0927 10:54:08.787337 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:08.787340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:08.787383 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:08.787450 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:08.788364 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:08.788902 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:08.790089 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:08.790299 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:08.790375 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:08.791463 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:08.791466 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:08.791468 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:08.791480 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:08.791498 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:08.791506 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:08.791567 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:08.791767 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 10:54:08.791825 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:54:08.792747 24311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:08.784737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:33995 (local address 127.14.68.3:39363)
0927 10:54:08.785510 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:08.785576 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:08.787146 (+ 1570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:08.787626 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:08.787633 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:08.787746 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:08.788242 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:08.788256 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:08.789014 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:08.789018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:08.789981 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:08.789991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:08.790429 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:08.790432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:08.790620 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:08.791224 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:54:08.791226 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:08.791263 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:54:08.791323 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:08.791583 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:08.791590 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:08.791593 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 10:54:08.791713 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 10:54:08.792015 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":143,"threads_started":1}
I20260927 10:54:08.793779 24309 heartbeater.cc:347] Connected to a master server at 127.14.68.62:33995
I20260927 10:54:08.794008 24309 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:08.794493 24309 heartbeater.cc:511] Master 127.14.68.62:33995 requested a full tablet report, sending...
I20260927 10:54:08.795609 23859 ts_manager.cc:194] Registered new tserver with Master: 4919661b97a247a1bbc8415a7de883bd (127.14.68.3:36143)
I20260927 10:54:08.796315 23859 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.68.3:39363
I20260927 10:54:08.798573 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 10:54:08.801556 23898 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:08.800634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46372 (local address 127.14.68.62:33995)
0927 10:54:08.800758 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:08.800763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:08.800944 (+ 181us) server_negotiation.cc:407] Connection header received
0927 10:54:08.801081 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:08.801084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:08.801129 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:08.801231 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:08.801468 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46372: BlockingRecv error: recv got EOF from 127.0.0.1:46372 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:54:08.801862 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 23919
I20260927 10:54:08.807379 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 24050
I20260927 10:54:08.814175 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 24181
I20260927 10:54:08.821130 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 23828
2026-09-27T10:54:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (840 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T10:54:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:08Z Disabled control of system clock
I20260927 10:54:08.850497 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:45055
--webserver_interface=127.14.68.62
--webserver_port=0
--builtin_ntp_servers=127.14.68.20:40425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:45055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 10:54:08.960402 24320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:08.960652 24320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:08.960718 24320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:08.964653 24320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:08.964751 24320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:08.964782 24320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:08.964810 24320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:08.964851 24320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:08.969623 24320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:40425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:45055
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:45055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:54:08.970548 24320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:08.971880 24320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:08.980510 24325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:08.980592 24320 server_base.cc:1034] running on GCE node
W20260927 10:54:08.980508 24328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:08.980502 24326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:08.981298 24320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:08.982431 24320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:08.983623 24320 hybrid_clock.cc:648] HybridClock initialized: now 1790506448983606 us; error 33 us; skew 500 ppm
I20260927 10:54:08.985497 24320 webserver.cc:460] Webserver started at http://127.14.68.62:42341/ using document root <none> and password file <none>
I20260927 10:54:08.986088 24320 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:08.986181 24320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:08.986395 24320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:08.988483 24320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "30365d3d8c7840c298f7ab6e10cd08f9"
format_stamp: "Formatted at 2026-09-27 10:54:08 on dist-test-slave-tz51"
server_key: "e740ab6c03d6ae2651e98b93dca0b46e"
server_key_iv: "c5a66de957683ee426d5e05d8dcf4333"
server_key_version: "encryptionkey@0"
I20260927 10:54:08.989040 24320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "30365d3d8c7840c298f7ab6e10cd08f9"
format_stamp: "Formatted at 2026-09-27 10:54:08 on dist-test-slave-tz51"
server_key: "e740ab6c03d6ae2651e98b93dca0b46e"
server_key_iv: "c5a66de957683ee426d5e05d8dcf4333"
server_key_version: "encryptionkey@0"
I20260927 10:54:08.993012 24320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:54:08.995355 24334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:08.996609 24320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:54:08.996760 24320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "30365d3d8c7840c298f7ab6e10cd08f9"
format_stamp: "Formatted at 2026-09-27 10:54:08 on dist-test-slave-tz51"
server_key: "e740ab6c03d6ae2651e98b93dca0b46e"
server_key_iv: "c5a66de957683ee426d5e05d8dcf4333"
server_key_version: "encryptionkey@0"
I20260927 10:54:08.996874 24320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:09.001543 24320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:09.002194 24320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:09.002382 24320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:09.009325 24320 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:45055
I20260927 10:54:09.009333 24386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:45055 every 8 connection(s)
I20260927 10:54:09.010203 24320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:09.013481 24387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:09.015729 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 24320
I20260927 10:54:09.015862 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:54:09.016172 14608 external_mini_cluster.cc:1428] Setting key cd6a814629fc840c7bc3a1b9f68a9e44
I20260927 10:54:09.019619 24387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30365d3d8c7840c298f7ab6e10cd08f9: Bootstrap starting.
I20260927 10:54:09.021685 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:09.017701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48604 (local address 127.14.68.62:45055)
0927 10:54:09.018344 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:09.018358 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:09.018391 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:54:09.019000 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:09.019016 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:09.019355 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:09.019684 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:09.020512 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:09.020517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:09.020529 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:09.020539 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:09.020591 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:09.020606 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:09.020754 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:09.020823 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 10:54:09.021046 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":131,"threads_started":1}
I20260927 10:54:09.022017 24387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30365d3d8c7840c298f7ab6e10cd08f9: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:09.022816 24387 log.cc:824] T 00000000000000000000000000000000 P 30365d3d8c7840c298f7ab6e10cd08f9: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:09.024874 24387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30365d3d8c7840c298f7ab6e10cd08f9: No bootstrap required, opened a new log
I20260927 10:54:09.028587 24387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30365d3d8c7840c298f7ab6e10cd08f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30365d3d8c7840c298f7ab6e10cd08f9" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45055 } }
I20260927 10:54:09.028791 24387 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30365d3d8c7840c298f7ab6e10cd08f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:09.028825 24387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30365d3d8c7840c298f7ab6e10cd08f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30365d3d8c7840c298f7ab6e10cd08f9, State: Initialized, Role: FOLLOWER
I20260927 10:54:09.029158 24387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30365d3d8c7840c298f7ab6e10cd08f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30365d3d8c7840c298f7ab6e10cd08f9" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45055 } }
I20260927 10:54:09.029268 24387 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30365d3d8c7840c298f7ab6e10cd08f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:09.029310 24387 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30365d3d8c7840c298f7ab6e10cd08f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:09.029383 24387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30365d3d8c7840c298f7ab6e10cd08f9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:09.030148 24387 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30365d3d8c7840c298f7ab6e10cd08f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30365d3d8c7840c298f7ab6e10cd08f9" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45055 } }
I20260927 10:54:09.030426 24387 leader_election.cc:302] T 00000000000000000000000000000000 P 30365d3d8c7840c298f7ab6e10cd08f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30365d3d8c7840c298f7ab6e10cd08f9; no voters:
I20260927 10:54:09.030654 24387 leader_election.cc:288] T 00000000000000000000000000000000 P 30365d3d8c7840c298f7ab6e10cd08f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:09.030735 24392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30365d3d8c7840c298f7ab6e10cd08f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:09.031004 24392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30365d3d8c7840c298f7ab6e10cd08f9 [term 1 LEADER]: Becoming Leader. State: Replica: 30365d3d8c7840c298f7ab6e10cd08f9, State: Running, Role: LEADER
I20260927 10:54:09.031334 24392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30365d3d8c7840c298f7ab6e10cd08f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30365d3d8c7840c298f7ab6e10cd08f9" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45055 } }
I20260927 10:54:09.031839 24387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30365d3d8c7840c298f7ab6e10cd08f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:09.033063 24393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30365d3d8c7840c298f7ab6e10cd08f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30365d3d8c7840c298f7ab6e10cd08f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30365d3d8c7840c298f7ab6e10cd08f9" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45055 } } }
I20260927 10:54:09.033280 24393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30365d3d8c7840c298f7ab6e10cd08f9 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:09.033324 24394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30365d3d8c7840c298f7ab6e10cd08f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30365d3d8c7840c298f7ab6e10cd08f9. Latest consensus state: current_term: 1 leader_uuid: "30365d3d8c7840c298f7ab6e10cd08f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30365d3d8c7840c298f7ab6e10cd08f9" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45055 } } }
I20260927 10:54:09.033500 24394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30365d3d8c7840c298f7ab6e10cd08f9 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:09.033691 24401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:09.036669 24401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:09.041584 24401 catalog_manager.cc:1347] Generated new cluster ID: 375f3ad86b28472d80ed82a7737c4615
I20260927 10:54:09.041649 24401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:09.077517 24401 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:09.077869 24401 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:09.087535 24401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30365d3d8c7840c298f7ab6e10cd08f9: Generated new TSK 0
I20260927 10:54:09.088337 24401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:54:09.091733 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:45055
--builtin_ntp_servers=127.14.68.20:40425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 10:54:09.206250 24411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:09.206537 24411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:09.206609 24411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:09.210537 24411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:09.210610 24411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:09.210711 24411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:54:09.215581 24411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:40425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:45055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:54:09.216662 24411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:09.218001 24411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:09.227605 24419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:09.227640 24416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:09.227607 24411 server_base.cc:1034] running on GCE node
W20260927 10:54:09.227607 24417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:09.228222 24411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:09.228857 24411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:09.230042 24411 hybrid_clock.cc:648] HybridClock initialized: now 1790506449230027 us; error 66 us; skew 500 ppm
I20260927 10:54:09.232082 24411 webserver.cc:460] Webserver started at http://127.14.68.1:43905/ using document root <none> and password file <none>
I20260927 10:54:09.232733 24411 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:09.232823 24411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:09.233042 24411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:09.235311 24411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "fe90f20e568e4bb3b39107bf9f91372d"
format_stamp: "Formatted at 2026-09-27 10:54:09 on dist-test-slave-tz51"
server_key: "815abcc561263460f90f19089e61831a"
server_key_iv: "e40f84fb975e2a387f425ba606fa733e"
server_key_version: "encryptionkey@0"
I20260927 10:54:09.235895 24411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "fe90f20e568e4bb3b39107bf9f91372d"
format_stamp: "Formatted at 2026-09-27 10:54:09 on dist-test-slave-tz51"
server_key: "815abcc561263460f90f19089e61831a"
server_key_iv: "e40f84fb975e2a387f425ba606fa733e"
server_key_version: "encryptionkey@0"
I20260927 10:54:09.239461 24411 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:54:09.241982 24425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:09.243252 24411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:09.243386 24411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "fe90f20e568e4bb3b39107bf9f91372d"
format_stamp: "Formatted at 2026-09-27 10:54:09 on dist-test-slave-tz51"
server_key: "815abcc561263460f90f19089e61831a"
server_key_iv: "e40f84fb975e2a387f425ba606fa733e"
server_key_version: "encryptionkey@0"
I20260927 10:54:09.243508 24411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:09.248086 24411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:09.248677 24411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:09.248862 24411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:09.249385 24411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:09.250267 24411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:09.250313 24411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:09.250349 24411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:09.250403 24411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:09.259398 24411 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:45803
I20260927 10:54:09.259428 24538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:45803 every 8 connection(s)
I20260927 10:54:09.260478 24411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:54:09.266831 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:09.262357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:38587 (local address 127.14.68.62:45055)
0927 10:54:09.262533 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:09.262537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:09.263570 (+ 1033us) server_negotiation.cc:407] Connection header received
0927 10:54:09.265030 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:09.265033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:09.265081 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:09.265182 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:09.266335 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:09.266339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:09.266341 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:09.266354 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:09.266377 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:09.266395 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:09.266497 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:09.266647 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 10:54:09.266693 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 10:54:09.267632 24541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:09.262713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:45055 (local address 127.14.68.1:38587)
0927 10:54:09.263406 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:09.263447 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:09.264799 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:09.265353 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:09.265359 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:09.265448 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:09.266051 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:54:09.266053 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:09.266106 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:54:09.266181 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:09.266478 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:09.266487 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:09.266492 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 10:54:09.266672 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 10:54:09.266904 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
I20260927 10:54:09.268122 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 24411
I20260927 10:54:09.268241 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:54:09.268543 14608 external_mini_cluster.cc:1428] Setting key ab7096ef4b0c1e4ad3253322b44ba930
I20260927 10:54:09.268947 24539 heartbeater.cc:347] Connected to a master server at 127.14.68.62:45055
I20260927 10:54:09.269210 24539 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:09.269601 24539 heartbeater.cc:511] Master 127.14.68.62:45055 requested a full tablet report, sending...
I20260927 10:54:09.270506 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:45055
--builtin_ntp_servers=127.14.68.20:40425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 10:54:09.271215 24351 ts_manager.cc:194] Registered new tserver with Master: fe90f20e568e4bb3b39107bf9f91372d (127.14.68.1:45803)
W20260927 10:54:09.384701 24542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:09.384935 24542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:09.384974 24542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:09.389030 24542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:09.389081 24542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:09.389195 24542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:54:09.393761 24542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:40425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:45055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:54:09.394757 24542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:09.396070 24542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:09.405980 24547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:09.405997 24548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:09.405987 24550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:09.406340 24542 server_base.cc:1034] running on GCE node
I20260927 10:54:09.406703 24542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:09.407305 24542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:09.408464 24542 hybrid_clock.cc:648] HybridClock initialized: now 1790506449408450 us; error 26 us; skew 500 ppm
I20260927 10:54:09.410243 24542 webserver.cc:460] Webserver started at http://127.14.68.2:42173/ using document root <none> and password file <none>
I20260927 10:54:09.410804 24542 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:09.410861 24542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:09.411098 24542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:09.413235 24542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "4f310e44eab5408abf1c4d0546aa86ed"
format_stamp: "Formatted at 2026-09-27 10:54:09 on dist-test-slave-tz51"
server_key: "b430967564b26684c44884a87cf454b1"
server_key_iv: "97a047d23ac83ed4ec3b252140d01120"
server_key_version: "encryptionkey@0"
I20260927 10:54:09.413798 24542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "4f310e44eab5408abf1c4d0546aa86ed"
format_stamp: "Formatted at 2026-09-27 10:54:09 on dist-test-slave-tz51"
server_key: "b430967564b26684c44884a87cf454b1"
server_key_iv: "97a047d23ac83ed4ec3b252140d01120"
server_key_version: "encryptionkey@0"
I20260927 10:54:09.417356 24542 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:54:09.419781 24556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:09.420990 24542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:09.421118 24542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "4f310e44eab5408abf1c4d0546aa86ed"
format_stamp: "Formatted at 2026-09-27 10:54:09 on dist-test-slave-tz51"
server_key: "b430967564b26684c44884a87cf454b1"
server_key_iv: "97a047d23ac83ed4ec3b252140d01120"
server_key_version: "encryptionkey@0"
I20260927 10:54:09.421234 24542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:09.426759 24542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:09.427348 24542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:09.427526 24542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:09.428112 24542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:09.429010 24542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:09.429088 24542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:09.429157 24542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:09.429190 24542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:09.437994 24542 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:46183
I20260927 10:54:09.438014 24669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:46183 every 8 connection(s)
I20260927 10:54:09.438861 24542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:54:09.445070 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:09.440743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:52419 (local address 127.14.68.62:45055)
0927 10:54:09.440882 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:09.440886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:09.441917 (+ 1031us) server_negotiation.cc:407] Connection header received
0927 10:54:09.443247 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:09.443251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:09.443302 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:09.443412 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:09.444586 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:09.444590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:09.444593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:09.444607 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:09.444637 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:09.444645 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:09.444745 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:09.444885 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 10:54:09.444932 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:54:09.445901 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:09.441085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:45055 (local address 127.14.68.2:52419)
0927 10:54:09.441769 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:09.441808 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:09.443013 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:09.443581 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:09.443588 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:09.443681 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:09.444329 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:54:09.444330 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:09.444370 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:54:09.444436 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:09.444736 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:09.444745 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:09.444751 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:54:09.444928 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 10:54:09.445162 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":132,"threads_started":1}
I20260927 10:54:09.446684 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 24542
I20260927 10:54:09.446810 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:54:09.447127 14608 external_mini_cluster.cc:1428] Setting key 9e1abc5f4e984caeee62ae8256de7e9b
I20260927 10:54:09.447144 24670 heartbeater.cc:347] Connected to a master server at 127.14.68.62:45055
I20260927 10:54:09.447377 24670 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:09.447867 24670 heartbeater.cc:511] Master 127.14.68.62:45055 requested a full tablet report, sending...
I20260927 10:54:09.448874 24351 ts_manager.cc:194] Registered new tserver with Master: 4f310e44eab5408abf1c4d0546aa86ed (127.14.68.2:46183)
I20260927 10:54:09.449144 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:45055
--builtin_ntp_servers=127.14.68.20:40425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 10:54:09.557302 24673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:09.557531 24673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:09.557592 24673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:09.562124 24673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:09.562204 24673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:09.562288 24673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:54:09.566964 24673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:40425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:45055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:54:09.568001 24673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:09.569312 24673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:09.578459 24679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:09.578471 24681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:09.578472 24678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:09.578560 24673 server_base.cc:1034] running on GCE node
I20260927 10:54:09.579057 24673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:09.579645 24673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:09.580844 24673 hybrid_clock.cc:648] HybridClock initialized: now 1790506449580832 us; error 24 us; skew 500 ppm
I20260927 10:54:09.582602 24673 webserver.cc:460] Webserver started at http://127.14.68.3:42073/ using document root <none> and password file <none>
I20260927 10:54:09.583148 24673 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:09.583215 24673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:09.583417 24673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:09.585398 24673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "e6e853a6f7e1403fbb084cb00e1ae4e6"
format_stamp: "Formatted at 2026-09-27 10:54:09 on dist-test-slave-tz51"
server_key: "7758eb13e260427a08a26f5f623ae0e1"
server_key_iv: "82e4b43aadb882d0261ea3c98f4c27ff"
server_key_version: "encryptionkey@0"
I20260927 10:54:09.585940 24673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "e6e853a6f7e1403fbb084cb00e1ae4e6"
format_stamp: "Formatted at 2026-09-27 10:54:09 on dist-test-slave-tz51"
server_key: "7758eb13e260427a08a26f5f623ae0e1"
server_key_iv: "82e4b43aadb882d0261ea3c98f4c27ff"
server_key_version: "encryptionkey@0"
I20260927 10:54:09.589728 24673 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:54:09.592136 24687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:09.593269 24673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:54:09.593395 24673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "e6e853a6f7e1403fbb084cb00e1ae4e6"
format_stamp: "Formatted at 2026-09-27 10:54:09 on dist-test-slave-tz51"
server_key: "7758eb13e260427a08a26f5f623ae0e1"
server_key_iv: "82e4b43aadb882d0261ea3c98f4c27ff"
server_key_version: "encryptionkey@0"
I20260927 10:54:09.593498 24673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:09.597790 24673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:09.598346 24673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:09.598529 24673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:09.599138 24673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:09.600067 24673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:09.600133 24673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:09.600207 24673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:09.600257 24673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:09.608996 24673 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:45321
I20260927 10:54:09.609063 24800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:45321 every 8 connection(s)
I20260927 10:54:09.609845 24673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:54:09.614880 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 24673
I20260927 10:54:09.615005 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:54:09.615306 14608 external_mini_cluster.cc:1428] Setting key 5d72c139c84a6850228845754810cacb
I20260927 10:54:09.615890 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:09.611537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:46997 (local address 127.14.68.62:45055)
0927 10:54:09.611661 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:09.611665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:09.612735 (+ 1070us) server_negotiation.cc:407] Connection header received
0927 10:54:09.614082 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:09.614085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:09.614121 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:09.614201 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:09.615319 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:09.615323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:09.615326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:09.615337 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:09.615358 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:09.615374 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:09.615483 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:09.615681 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 10:54:09.615743 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:54:09.616838 24803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:09.611986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:45055 (local address 127.14.68.3:46997)
0927 10:54:09.612601 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:09.612637 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:09.613892 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:09.614335 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:09.614341 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:09.614422 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:09.615015 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:54:09.615017 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:09.615071 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:54:09.615178 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:09.615495 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:09.615504 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:09.615510 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:54:09.615728 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 10:54:09.615973 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":114,"threads_started":1}
I20260927 10:54:09.617826 24801 heartbeater.cc:347] Connected to a master server at 127.14.68.62:45055
I20260927 10:54:09.618039 24801 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:09.618412 24801 heartbeater.cc:511] Master 127.14.68.62:45055 requested a full tablet report, sending...
I20260927 10:54:09.619328 24351 ts_manager.cc:194] Registered new tserver with Master: e6e853a6f7e1403fbb084cb00e1ae4e6 (127.14.68.3:45321)
I20260927 10:54:09.628861 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 10:54:09.631544 24390 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:09.630744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48612 (local address 127.14.68.62:45055)
0927 10:54:09.630908 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:09.630911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:09.631019 (+ 108us) server_negotiation.cc:407] Connection header received
0927 10:54:09.631109 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:09.631111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:09.631137 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:09.631197 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:09.631430 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48612: BlockingRecv error: recv got EOF from 127.0.0.1:48612 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 10:54:09.631717 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 24411
I20260927 10:54:09.637511 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 24542
I20260927 10:54:09.645925 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 24673
I20260927 10:54:09.653316 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 24320
2026-09-27T10:54:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (833 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:09 dist-test-slave-tz51 krb5kdc[24811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:09 dist-test-slave-tz51 krb5kdc[24811](info): set up 2 sockets
Sep 27 10:54:09 dist-test-slave-tz51 krb5kdc[24811](info): commencing operation
krb5kdc: starting...
W20260927 10:54:10.692745 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:54:10 dist-test-slave-tz51 krb5kdc[24811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:54:10.814446 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:46467
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:38865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:46467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:10.923291 24824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:10.923566 24824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:10.923607 24824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:10.927661 24824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:10.927764 24824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:10.927783 24824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:10.927801 24824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:10.927816 24824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:10.933318 24824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:38865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:46467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:46467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:54:10.934438 24824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:10.935487 24824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:10.945266 24832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:10.945276 24829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:10.945261 24830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:10.945634 24824 server_base.cc:1034] running on GCE node
I20260927 10:54:10.946364 24824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:10.947516 24824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:10.948736 24824 hybrid_clock.cc:648] HybridClock initialized: now 1790506450948700 us; error 52 us; skew 500 ppm
Sep 27 10:54:10 dist-test-slave-tz51 krb5kdc[24811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506450, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:10.951570 24824 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:54:10.952513 24824 webserver.cc:460] Webserver started at http://127.14.68.62:42903/ using document root <none> and password file <none>
I20260927 10:54:10.953079 24824 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:10.953176 24824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:10.953378 24824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:10.955354 24824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "881220d313824adaa13abcbd415014c5"
format_stamp: "Formatted at 2026-09-27 10:54:10 on dist-test-slave-tz51"
server_key: "d623c91a5f9539f085199f53fc91c805"
server_key_iv: "761f12028c323a03a94e74b40d0f62c2"
server_key_version: "encryptionkey@0"
I20260927 10:54:10.955952 24824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "881220d313824adaa13abcbd415014c5"
format_stamp: "Formatted at 2026-09-27 10:54:10 on dist-test-slave-tz51"
server_key: "d623c91a5f9539f085199f53fc91c805"
server_key_iv: "761f12028c323a03a94e74b40d0f62c2"
server_key_version: "encryptionkey@0"
I20260927 10:54:10.959517 24824 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:54:10.962216 24839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:10.963405 24824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:10.963543 24824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "881220d313824adaa13abcbd415014c5"
format_stamp: "Formatted at 2026-09-27 10:54:10 on dist-test-slave-tz51"
server_key: "d623c91a5f9539f085199f53fc91c805"
server_key_iv: "761f12028c323a03a94e74b40d0f62c2"
server_key_version: "encryptionkey@0"
I20260927 10:54:10.963657 24824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:10.979285 24824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:10.982688 24824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:10.982914 24824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:10.990478 24824 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:46467
I20260927 10:54:10.990875 24891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:46467 every 8 connection(s)
I20260927 10:54:10.991562 24824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:10.994421 24892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:10.999871 24892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5: Bootstrap starting.
I20260927 10:54:11.001319 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 24824
I20260927 10:54:11.001478 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:54:11.001786 14608 external_mini_cluster.cc:1428] Setting key fc09e33075bf13daaf33b579d6bbe22f
I20260927 10:54:11.002501 24892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:11.003280 24892 log.cc:824] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:11.005334 24892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5: No bootstrap required, opened a new log
I20260927 10:54:11.009425 24892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881220d313824adaa13abcbd415014c5" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46467 } }
I20260927 10:54:11.009727 24892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:11.009783 24892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881220d313824adaa13abcbd415014c5, State: Initialized, Role: FOLLOWER
I20260927 10:54:11.010298 24892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881220d313824adaa13abcbd415014c5" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46467 } }
I20260927 10:54:11.010454 24892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:11.010528 24892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:11.010646 24892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 10:54:11 dist-test-slave-tz51 krb5kdc[24811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:54:11.011586 24892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881220d313824adaa13abcbd415014c5" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46467 } }
I20260927 10:54:11.011945 24892 leader_election.cc:302] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 881220d313824adaa13abcbd415014c5; no voters:
I20260927 10:54:11.012295 24892 leader_election.cc:288] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:11.012362 24897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:11.012568 24897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5 [term 1 LEADER]: Becoming Leader. State: Replica: 881220d313824adaa13abcbd415014c5, State: Running, Role: LEADER
I20260927 10:54:11.012894 24897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881220d313824adaa13abcbd415014c5" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46467 } }
I20260927 10:54:11.013556 24892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:11.014781 24899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 881220d313824adaa13abcbd415014c5. Latest consensus state: current_term: 1 leader_uuid: "881220d313824adaa13abcbd415014c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881220d313824adaa13abcbd415014c5" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46467 } } }
I20260927 10:54:11.015177 24899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:11.015080 24898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "881220d313824adaa13abcbd415014c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881220d313824adaa13abcbd415014c5" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46467 } } }
I20260927 10:54:11.015482 24898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:11.015889 24911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:11.017290 24895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:11.002990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38766 (local address 127.14.68.62:46467)
0927 10:54:11.003472 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:11.003489 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:11.003525 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:54:11.004410 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:11.004431 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:11.004803 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:11.005100 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:11.006036 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.007072 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.008374 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.008829 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.008946 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:11.011758 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:11.011765 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:11.011781 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:11.011818 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:11.013796 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:11.014298 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:11.014306 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:11.014314 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:11.014476 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:11.014819 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:11.014822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:11.014824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:11.015377 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:11.015594 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:11.016068 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 10:54:11.016403 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":166,"threads_started":1}
I20260927 10:54:11.019106 24911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:11.024315 24911 catalog_manager.cc:1347] Generated new cluster ID: 47293502eaee45cdb8e61d66979cd2c3
I20260927 10:54:11.024400 24911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:11.032264 24911 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:11.033322 24911 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:11.045403 24911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5: Generated new TSK 0
I20260927 10:54:11.046156 24911 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
I20260927 10:54:11.078325 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46467
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:38865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:11.187611 24918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:11.187907 24918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:11.187949 24918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:11.191852 24918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:11.191905 24918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:11.191989 24918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:54:11.197124 24918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:38865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:54:11.198112 24918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:11.199116 24918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:11.208556 24923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:11.208626 24926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:11.208634 24918 server_base.cc:1034] running on GCE node
W20260927 10:54:11.208670 24924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:11.209131 24918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:11.209741 24918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:11.210925 24918 hybrid_clock.cc:648] HybridClock initialized: now 1790506451210913 us; error 26 us; skew 500 ppm
Sep 27 10:54:11 dist-test-slave-tz51 krb5kdc[24811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506451, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:11.213593 24918 init.cc:377] Logged in from keytab as kudu/127.14.68.1@KRBTEST.COM (short username kudu)
I20260927 10:54:11.214488 24918 webserver.cc:460] Webserver started at http://127.14.68.1:42469/ using document root <none> and password file <none>
I20260927 10:54:11.215020 24918 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:11.215106 24918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:11.215313 24918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:11.217542 24918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "fd9c1d91649d4b5d9225b49a3ba8940c"
format_stamp: "Formatted at 2026-09-27 10:54:11 on dist-test-slave-tz51"
server_key: "d1273db96443cf7c69ae556025f3dd6c"
server_key_iv: "1c09a1629e2d1b78097b5482dab9240a"
server_key_version: "encryptionkey@0"
I20260927 10:54:11.218102 24918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "fd9c1d91649d4b5d9225b49a3ba8940c"
format_stamp: "Formatted at 2026-09-27 10:54:11 on dist-test-slave-tz51"
server_key: "d1273db96443cf7c69ae556025f3dd6c"
server_key_iv: "1c09a1629e2d1b78097b5482dab9240a"
server_key_version: "encryptionkey@0"
I20260927 10:54:11.221688 24918 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:54:11.223994 24933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:11.225102 24918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:54:11.225236 24918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "fd9c1d91649d4b5d9225b49a3ba8940c"
format_stamp: "Formatted at 2026-09-27 10:54:11 on dist-test-slave-tz51"
server_key: "d1273db96443cf7c69ae556025f3dd6c"
server_key_iv: "1c09a1629e2d1b78097b5482dab9240a"
server_key_version: "encryptionkey@0"
I20260927 10:54:11.225338 24918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:11.243067 24918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:11.245914 24918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:11.246134 24918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:11.246721 24918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:11.247532 24918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:11.247611 24918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:11.247725 24918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:11.247778 24918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:11.256708 24918 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:42585
I20260927 10:54:11.256731 25046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:42585 every 8 connection(s)
I20260927 10:54:11.257619 24918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:54:11.264252 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 24918
I20260927 10:54:11.264381 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:54:11.264699 14608 external_mini_cluster.cc:1428] Setting key fb0d17934e69e55643847f4a0fd9f746
Sep 27 10:54:11 dist-test-slave-tz51 krb5kdc[24811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506451, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:54:11.271929 24895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:11.259425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:38905 (local address 127.14.68.62:46467)
0927 10:54:11.259574 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:11.259579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:11.260439 (+ 860us) server_negotiation.cc:407] Connection header received
0927 10:54:11.261735 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:11.261738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:11.261783 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:11.261856 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:11.262983 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.263510 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.264786 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.265111 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.265200 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:11.268255 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:11.268259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:11.268261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:11.268284 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:11.269662 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:11.270198 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:11.270203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:11.270204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:11.270293 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:11.270737 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:11.270740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:11.270742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:11.270939 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:11.271049 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:11.271570 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 10:54:11.271744 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:54:11.272758 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:11.259836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:46467 (local address 127.14.68.1:38905)
0927 10:54:11.260282 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:11.260324 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:11.261545 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:11.262026 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:11.262032 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:11.262393 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:11.262852 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:11.262866 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.263614 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.263617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:11.264664 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:11.264673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.265280 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.265285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:11.265465 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:11.266109 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:11.266131 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:11.268071 (+ 1940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:11.269795 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:11.269801 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:11.269818 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:11.270084 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:11.270486 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:11.270488 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:11.270489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:11.270625 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:11.271048 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:11.271056 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:11.271321 (+ 265us) client_negotiation.cc:769] Sending connection context
0927 10:54:11.271513 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 10:54:11.271881 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":112,"threads_started":1}
I20260927 10:54:11.274212 25047 heartbeater.cc:347] Connected to a master server at 127.14.68.62:46467
WARNING: no policy specified for kudu/127.14.68.2@KRBTEST.COM; defaulting to no policy
I20260927 10:54:11.274466 25047 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:11.274983 25047 heartbeater.cc:511] Master 127.14.68.62:46467 requested a full tablet report, sending...
I20260927 10:54:11.276615 24856 ts_manager.cc:194] Registered new tserver with Master: fd9c1d91649d4b5d9225b49a3ba8940c (127.14.68.1:42585)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.2@KRBTEST.COM" created.
I20260927 10:54:11.278115 24856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:38905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
I20260927 10:54:11.294052 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46467
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:38865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:11.400745 25052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:11.400993 25052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:11.401036 25052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:11.404966 25052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:11.405018 25052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:11.405073 25052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:54:11.410072 25052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:38865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:54:11.411043 25052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:11.412057 25052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:11.421721 25058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:11.421784 25057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:11.421721 25060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:11.421832 25052 server_base.cc:1034] running on GCE node
I20260927 10:54:11.422333 25052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:11.422941 25052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:11.424093 25052 hybrid_clock.cc:648] HybridClock initialized: now 1790506451424081 us; error 25 us; skew 500 ppm
Sep 27 10:54:11 dist-test-slave-tz51 krb5kdc[24811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506451, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:11.426695 25052 init.cc:377] Logged in from keytab as kudu/127.14.68.2@KRBTEST.COM (short username kudu)
I20260927 10:54:11.427523 25052 webserver.cc:460] Webserver started at http://127.14.68.2:39267/ using document root <none> and password file <none>
I20260927 10:54:11.428122 25052 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:11.428184 25052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:11.428344 25052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:11.430267 25052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "ccd30d31bfc34101aaa48a6c93837e20"
format_stamp: "Formatted at 2026-09-27 10:54:11 on dist-test-slave-tz51"
server_key: "0232f7afb75f01e935a454bcfe3092dc"
server_key_iv: "bf0d666d8f109ea68c4750e14205dc1f"
server_key_version: "encryptionkey@0"
I20260927 10:54:11.430768 25052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "ccd30d31bfc34101aaa48a6c93837e20"
format_stamp: "Formatted at 2026-09-27 10:54:11 on dist-test-slave-tz51"
server_key: "0232f7afb75f01e935a454bcfe3092dc"
server_key_iv: "bf0d666d8f109ea68c4750e14205dc1f"
server_key_version: "encryptionkey@0"
I20260927 10:54:11.434284 25052 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:54:11.436544 25067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:11.437698 25052 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:54:11.437844 25052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "ccd30d31bfc34101aaa48a6c93837e20"
format_stamp: "Formatted at 2026-09-27 10:54:11 on dist-test-slave-tz51"
server_key: "0232f7afb75f01e935a454bcfe3092dc"
server_key_iv: "bf0d666d8f109ea68c4750e14205dc1f"
server_key_version: "encryptionkey@0"
I20260927 10:54:11.437953 25052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:11.455060 25052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:11.457641 25052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:11.457835 25052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:11.458377 25052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:11.459173 25052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:11.459251 25052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:11.459322 25052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:11.459378 25052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:11.468207 25052 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:35977
I20260927 10:54:11.468242 25180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:35977 every 8 connection(s)
I20260927 10:54:11.469079 25052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:54:11.469689 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 25052
I20260927 10:54:11.469810 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:54:11.470134 14608 external_mini_cluster.cc:1428] Setting key 2818dd859d752bc31f8e7e96d41ab8f6
WARNING: no policy specified for kudu/127.14.68.3@KRBTEST.COM; defaulting to no policy
Sep 27 10:54:11 dist-test-slave-tz51 krb5kdc[24811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506451, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.3@KRBTEST.COM" created.
I20260927 10:54:11.485335 24895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:11.470954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:36875 (local address 127.14.68.62:46467)
0927 10:54:11.471098 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:11.471103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:11.472241 (+ 1138us) server_negotiation.cc:407] Connection header received
0927 10:54:11.474000 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:11.474005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:11.474062 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:11.474156 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:11.475556 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.476198 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.478032 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.478240 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.478320 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:11.481071 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:11.481074 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:11.481076 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:11.481099 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:11.482628 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:11.483786 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:11.483789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:11.483790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:11.483866 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:11.484214 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:11.484217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:11.484219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:11.484403 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:11.484511 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:11.484993 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 10:54:11.485139 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:54:11.486127 25183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:11.471477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:46467 (local address 127.14.68.2:36875)
0927 10:54:11.472072 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:11.472131 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:11.473805 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:11.474300 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:11.474306 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:11.474766 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:11.475409 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:11.475431 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.476326 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.476330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:11.477901 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:11.477912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.478397 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.478402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:11.478611 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:11.479226 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:11.479246 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:11.480911 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:11.482728 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:11.482732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:11.482745 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:11.483127 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:11.483971 (+ 844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:11.483974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:11.483976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:11.484113 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:11.484509 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:11.484516 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:11.484759 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 10:54:11.484951 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 10:54:11.485275 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":184,"threads_started":1}
I20260927 10:54:11.487279 25181 heartbeater.cc:347] Connected to a master server at 127.14.68.62:46467
I20260927 10:54:11.487556 25181 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:11.488221 25181 heartbeater.cc:511] Master 127.14.68.62:46467 requested a full tablet report, sending...
I20260927 10:54:11.489418 24856 ts_manager.cc:194] Registered new tserver with Master: ccd30d31bfc34101aaa48a6c93837e20 (127.14.68.2:35977)
I20260927 10:54:11.490248 24856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:36875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
I20260927 10:54:11.498894 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46467
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:38865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:11.610057 25186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:11.610306 25186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:11.610348 25186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:11.614512 25186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:11.614568 25186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:11.614625 25186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:54:11.619899 25186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:38865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:54:11.620841 25186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:11.621774 25186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:11.631228 25191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:11.631436 25186 server_base.cc:1034] running on GCE node
W20260927 10:54:11.631253 25192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:11.631582 25194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:11.632005 25186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:11.632616 25186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:11.633808 25186 hybrid_clock.cc:648] HybridClock initialized: now 1790506451633794 us; error 29 us; skew 500 ppm
Sep 27 10:54:11 dist-test-slave-tz51 krb5kdc[24811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506451, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:11.636445 25186 init.cc:377] Logged in from keytab as kudu/127.14.68.3@KRBTEST.COM (short username kudu)
I20260927 10:54:11.637272 25186 webserver.cc:460] Webserver started at http://127.14.68.3:33023/ using document root <none> and password file <none>
I20260927 10:54:11.637802 25186 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:11.637888 25186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:11.638128 25186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:11.640115 25186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "a6fe48d72aba419e91f1811af8b32ea2"
format_stamp: "Formatted at 2026-09-27 10:54:11 on dist-test-slave-tz51"
server_key: "4f05523124822a36ea6dbec8fd805413"
server_key_iv: "d907bdf0b490337b1bccbd07a1f8236c"
server_key_version: "encryptionkey@0"
I20260927 10:54:11.640724 25186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "a6fe48d72aba419e91f1811af8b32ea2"
format_stamp: "Formatted at 2026-09-27 10:54:11 on dist-test-slave-tz51"
server_key: "4f05523124822a36ea6dbec8fd805413"
server_key_iv: "d907bdf0b490337b1bccbd07a1f8236c"
server_key_version: "encryptionkey@0"
I20260927 10:54:11.644448 25186 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:54:11.646863 25201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:11.648213 25186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:54:11.648344 25186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "a6fe48d72aba419e91f1811af8b32ea2"
format_stamp: "Formatted at 2026-09-27 10:54:11 on dist-test-slave-tz51"
server_key: "4f05523124822a36ea6dbec8fd805413"
server_key_iv: "d907bdf0b490337b1bccbd07a1f8236c"
server_key_version: "encryptionkey@0"
I20260927 10:54:11.648458 25186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:11.663337 25186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:11.666559 25186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:11.666764 25186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:11.667382 25186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:11.668251 25186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:11.668299 25186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:11.668339 25186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:11.668354 25186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:11.677143 25186 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:38211
I20260927 10:54:11.677167 25314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:38211 every 8 connection(s)
I20260927 10:54:11.677977 25186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:54:11.684947 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 25186
I20260927 10:54:11.685051 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:54:11.685398 14608 external_mini_cluster.cc:1428] Setting key 652f781b0ea8001cc04794e2d7aa7e39
Sep 27 10:54:11 dist-test-slave-tz51 krb5kdc[24811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506451, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:54:11.695832 24895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:11.679759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:43989 (local address 127.14.68.62:46467)
0927 10:54:11.679889 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:11.679893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:11.680707 (+ 814us) server_negotiation.cc:407] Connection header received
0927 10:54:11.681959 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:11.681961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:11.682037 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:11.682109 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:11.683242 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.683751 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.684892 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.685173 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.685248 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:11.688081 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:11.688084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:11.688086 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:11.688109 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:11.693103 (+ 4994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:11.693897 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:11.693903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:11.693905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:11.694022 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:11.694445 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:11.694448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:11.694450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:11.694731 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:11.694850 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:11.695437 (+ 587us) server_negotiation.cc:299] Negotiation successful
0927 10:54:11.695605 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:54:11.696835 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:11.680117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:46467 (local address 127.14.68.3:43989)
0927 10:54:11.680558 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:11.680601 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:11.681775 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:11.682284 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:11.682290 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:11.682634 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:11.683092 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:11.683105 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.683843 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.683846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:11.684779 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:11.684792 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.685324 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.685330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:11.685521 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:11.686161 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:11.686185 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:11.687945 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:11.693241 (+ 5296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:11.693262 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:11.693281 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:11.693745 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:11.694117 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:11.694123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:11.694127 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:11.694337 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:11.694846 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:11.694856 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:11.695173 (+ 317us) client_negotiation.cc:769] Sending connection context
0927 10:54:11.695391 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 10:54:11.695776 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":99,"threads_started":1}
I20260927 10:54:11.698065 25315 heartbeater.cc:347] Connected to a master server at 127.14.68.62:46467
I20260927 10:54:11.698309 25315 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:11.698868 25315 heartbeater.cc:511] Master 127.14.68.62:46467 requested a full tablet report, sending...
I20260927 10:54:11.699999 24856 ts_manager.cc:194] Registered new tserver with Master: a6fe48d72aba419e91f1811af8b32ea2 (127.14.68.3:38211)
I20260927 10:54:11.700611 24856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:43989
I20260927 10:54:11.711340 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:54:11.721064 24895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:11.713399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38782 (local address 127.14.68.62:46467)
0927 10:54:11.713555 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:11.713560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:11.713750 (+ 190us) server_negotiation.cc:407] Connection header received
0927 10:54:11.713910 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:11.713913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:11.713963 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:11.714045 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:11.714825 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.715354 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.716318 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.716547 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.716657 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:11.718126 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:11.718130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:11.718133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:11.718165 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:11.719355 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:11.719804 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:11.719807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:11.719809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:11.719896 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:11.720170 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:11.720175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:11.720176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:11.720402 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:11.720547 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:11.720780 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 10:54:11.720898 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:54:11.723815 24856 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:54:11.726716 24856 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:54:11.740437 25330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:11.735554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49020 (local address 127.14.68.3:38211)
0927 10:54:11.735968 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:11.735980 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:11.736002 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:54:11.736069 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:11.736073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:11.736259 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:11.736392 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:11.736775 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.737722 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.738882 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.739761 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.739883 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:11.740035 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:11.740182 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 10:54:11.740284 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":171,"threads_started":1}
I20260927 10:54:11.740434 25326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:11.734790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:38211 (local address 127.0.0.1:49020)
0927 10:54:11.735224 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:11.735236 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:11.735294 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:11.736540 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:11.736543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:11.736560 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:11.736641 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:11.736648 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.737842 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.737847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:11.738754 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:11.738763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.739886 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.739890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:11.739992 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:11.740027 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 10:54:11.740140 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 10:54:11.740276 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":25,"thread_start_us":90,"threads_started":1}
I20260927 10:54:11.740434 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:11.734584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:35977 (local address 127.0.0.1:57936)
0927 10:54:11.735106 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:11.735154 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:11.735281 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:11.735895 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:11.735902 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:11.735926 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:11.736057 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:11.736064 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.737430 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.737434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:11.738828 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:11.738837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.740100 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.740104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:11.740189 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:11.740209 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:54:11.740272 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 10:54:11.740327 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":94,"thread_start_us":70,"threads_started":1}
I20260927 10:54:11.741138 25327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:11.734736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57936 (local address 127.14.68.2:35977)
0927 10:54:11.735342 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:11.735357 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:11.735383 (+ 26us) server_negotiation.cc:407] Connection header received
0927 10:54:11.735465 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:11.735472 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:11.735688 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:11.735880 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:11.736187 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.737322 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.738962 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.739788 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.739905 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:11.740039 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:11.740499 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 10:54:11.740986 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":122,"threads_started":1}
I20260927 10:54:11.741432 25329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:11.735270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:42585 (local address 127.0.0.1:45322)
0927 10:54:11.736535 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:11.736549 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:11.736703 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:11.737141 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:11.737144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:11.737159 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:11.737271 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:11.737280 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.738611 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.738614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:11.739787 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:11.739799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.740980 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.740984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:11.741104 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:11.741129 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:54:11.741221 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 10:54:11.741291 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1207,"mutex_wait_us":4,"thread_start_us":88,"threads_started":1}
I20260927 10:54:11.742228 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:11.736067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45322 (local address 127.14.68.1:42585)
0927 10:54:11.736395 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:11.736411 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:11.736622 (+ 211us) server_negotiation.cc:407] Connection header received
0927 10:54:11.736884 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:11.736887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:11.737011 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:11.737150 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:11.737394 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.738476 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.739917 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.740819 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.740916 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:11.741046 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:11.741972 (+ 926us) server_negotiation.cc:299] Negotiation successful
0927 10:54:11.742071 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":65,"threads_started":1}
I20260927 10:54:11.743614 25115 tablet_service.cc:1467] Processing CreateTablet for tablet f8428375925249f6b7e16112190c1909 (DEFAULT_TABLE table=test-table [id=9141848e1b944812b8dc5795b8a0f2f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:11.743614 25249 tablet_service.cc:1467] Processing CreateTablet for tablet f8428375925249f6b7e16112190c1909 (DEFAULT_TABLE table=test-table [id=9141848e1b944812b8dc5795b8a0f2f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:11.744663 25249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8428375925249f6b7e16112190c1909. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:11.744684 25115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8428375925249f6b7e16112190c1909. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:11.746187 24981 tablet_service.cc:1467] Processing CreateTablet for tablet f8428375925249f6b7e16112190c1909 (DEFAULT_TABLE table=test-table [id=9141848e1b944812b8dc5795b8a0f2f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:11.747112 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8428375925249f6b7e16112190c1909. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:11.751354 25332 tablet_bootstrap.cc:492] T f8428375925249f6b7e16112190c1909 P a6fe48d72aba419e91f1811af8b32ea2: Bootstrap starting.
I20260927 10:54:11.753613 25332 tablet_bootstrap.cc:654] T f8428375925249f6b7e16112190c1909 P a6fe48d72aba419e91f1811af8b32ea2: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:11.753607 25334 tablet_bootstrap.cc:492] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c: Bootstrap starting.
I20260927 10:54:11.754124 25333 tablet_bootstrap.cc:492] T f8428375925249f6b7e16112190c1909 P ccd30d31bfc34101aaa48a6c93837e20: Bootstrap starting.
I20260927 10:54:11.754913 25332 log.cc:824] T f8428375925249f6b7e16112190c1909 P a6fe48d72aba419e91f1811af8b32ea2: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:11.755592 25334 tablet_bootstrap.cc:654] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:11.755903 25333 tablet_bootstrap.cc:654] T f8428375925249f6b7e16112190c1909 P ccd30d31bfc34101aaa48a6c93837e20: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:11.756837 25334 log.cc:824] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:11.757089 25332 tablet_bootstrap.cc:492] T f8428375925249f6b7e16112190c1909 P a6fe48d72aba419e91f1811af8b32ea2: No bootstrap required, opened a new log
I20260927 10:54:11.757205 25333 log.cc:824] T f8428375925249f6b7e16112190c1909 P ccd30d31bfc34101aaa48a6c93837e20: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:11.757287 25332 ts_tablet_manager.cc:1397] T f8428375925249f6b7e16112190c1909 P a6fe48d72aba419e91f1811af8b32ea2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:54:11.758654 25334 tablet_bootstrap.cc:492] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c: No bootstrap required, opened a new log
I20260927 10:54:11.758766 25333 tablet_bootstrap.cc:492] T f8428375925249f6b7e16112190c1909 P ccd30d31bfc34101aaa48a6c93837e20: No bootstrap required, opened a new log
I20260927 10:54:11.758854 25334 ts_tablet_manager.cc:1397] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:54:11.758939 25333 ts_tablet_manager.cc:1397] T f8428375925249f6b7e16112190c1909 P ccd30d31bfc34101aaa48a6c93837e20: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 10:54:11.761246 25332 raft_consensus.cc:348] T f8428375925249f6b7e16112190c1909 P a6fe48d72aba419e91f1811af8b32ea2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd30d31bfc34101aaa48a6c93837e20" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 35977 } } peers { permanent_uuid: "a6fe48d72aba419e91f1811af8b32ea2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 38211 } } peers { permanent_uuid: "fd9c1d91649d4b5d9225b49a3ba8940c" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 42585 } }
I20260927 10:54:11.761511 25332 raft_consensus.cc:374] T f8428375925249f6b7e16112190c1909 P a6fe48d72aba419e91f1811af8b32ea2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:11.761601 25332 raft_consensus.cc:729] T f8428375925249f6b7e16112190c1909 P a6fe48d72aba419e91f1811af8b32ea2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6fe48d72aba419e91f1811af8b32ea2, State: Initialized, Role: FOLLOWER
I20260927 10:54:11.761643 25333 raft_consensus.cc:348] T f8428375925249f6b7e16112190c1909 P ccd30d31bfc34101aaa48a6c93837e20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd30d31bfc34101aaa48a6c93837e20" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 35977 } } peers { permanent_uuid: "a6fe48d72aba419e91f1811af8b32ea2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 38211 } } peers { permanent_uuid: "fd9c1d91649d4b5d9225b49a3ba8940c" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 42585 } }
I20260927 10:54:11.761914 25333 raft_consensus.cc:374] T f8428375925249f6b7e16112190c1909 P ccd30d31bfc34101aaa48a6c93837e20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:11.761974 25333 raft_consensus.cc:729] T f8428375925249f6b7e16112190c1909 P ccd30d31bfc34101aaa48a6c93837e20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccd30d31bfc34101aaa48a6c93837e20, State: Initialized, Role: FOLLOWER
I20260927 10:54:11.762059 25332 consensus_queue.cc:260] T f8428375925249f6b7e16112190c1909 P a6fe48d72aba419e91f1811af8b32ea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd30d31bfc34101aaa48a6c93837e20" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 35977 } } peers { permanent_uuid: "a6fe48d72aba419e91f1811af8b32ea2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 38211 } } peers { permanent_uuid: "fd9c1d91649d4b5d9225b49a3ba8940c" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 42585 } }
I20260927 10:54:11.762373 25333 consensus_queue.cc:260] T f8428375925249f6b7e16112190c1909 P ccd30d31bfc34101aaa48a6c93837e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd30d31bfc34101aaa48a6c93837e20" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 35977 } } peers { permanent_uuid: "a6fe48d72aba419e91f1811af8b32ea2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 38211 } } peers { permanent_uuid: "fd9c1d91649d4b5d9225b49a3ba8940c" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 42585 } }
I20260927 10:54:11.762535 25334 raft_consensus.cc:348] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd30d31bfc34101aaa48a6c93837e20" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 35977 } } peers { permanent_uuid: "a6fe48d72aba419e91f1811af8b32ea2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 38211 } } peers { permanent_uuid: "fd9c1d91649d4b5d9225b49a3ba8940c" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 42585 } }
I20260927 10:54:11.762837 25315 heartbeater.cc:503] Master 127.14.68.62:46467 was elected leader, sending a full tablet report...
I20260927 10:54:11.762882 25334 raft_consensus.cc:374] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:11.762933 25334 raft_consensus.cc:729] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd9c1d91649d4b5d9225b49a3ba8940c, State: Initialized, Role: FOLLOWER
I20260927 10:54:11.763151 25332 ts_tablet_manager.cc:1428] T f8428375925249f6b7e16112190c1909 P a6fe48d72aba419e91f1811af8b32ea2: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260927 10:54:11.763298 25333 ts_tablet_manager.cc:1428] T f8428375925249f6b7e16112190c1909 P ccd30d31bfc34101aaa48a6c93837e20: Time spent starting tablet: real 0.004s user 0.000s sys 0.005s
I20260927 10:54:11.763334 25334 consensus_queue.cc:260] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd30d31bfc34101aaa48a6c93837e20" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 35977 } } peers { permanent_uuid: "a6fe48d72aba419e91f1811af8b32ea2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 38211 } } peers { permanent_uuid: "fd9c1d91649d4b5d9225b49a3ba8940c" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 42585 } }
I20260927 10:54:11.763633 25181 heartbeater.cc:503] Master 127.14.68.62:46467 was elected leader, sending a full tablet report...
I20260927 10:54:11.765118 25334 ts_tablet_manager.cc:1428] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:54:11.765316 25047 heartbeater.cc:503] Master 127.14.68.62:46467 was elected leader, sending a full tablet report...
I20260927 10:54:11.844892 25340 raft_consensus.cc:482] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:54:11.845091 25340 raft_consensus.cc:504] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd30d31bfc34101aaa48a6c93837e20" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 35977 } } peers { permanent_uuid: "a6fe48d72aba419e91f1811af8b32ea2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 38211 } } peers { permanent_uuid: "fd9c1d91649d4b5d9225b49a3ba8940c" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 42585 } }
I20260927 10:54:11.846343 25340 leader_election.cc:288] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccd30d31bfc34101aaa48a6c93837e20 (127.14.68.2:35977), a6fe48d72aba419e91f1811af8b32ea2 (127.14.68.3:38211)
I20260927 10:54:11.850375 25330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:11.846741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:49217 (local address 127.14.68.3:38211)
0927 10:54:11.846934 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:11.846939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:11.847120 (+ 181us) server_negotiation.cc:407] Connection header received
0927 10:54:11.847291 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:11.847299 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:11.847342 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:11.847426 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:11.847725 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.848229 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.849400 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.849793 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.849904 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:11.849965 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:11.850163 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 10:54:11.850230 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 10:54:11.850376 25327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:11.846766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:45121 (local address 127.14.68.2:35977)
0927 10:54:11.846910 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:11.846914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:11.847173 (+ 259us) server_negotiation.cc:407] Connection header received
0927 10:54:11.847347 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:11.847351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:11.847399 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:11.847483 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:11.847764 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.848259 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.849409 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.849804 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.849921 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:11.849981 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:11.850176 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 10:54:11.850247 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:54:11.850420 25342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:11.846693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:35977 (local address 127.14.68.1:45121)
0927 10:54:11.847081 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:11.847092 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:11.847242 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:11.847494 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:11.847496 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:11.847511 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:11.847636 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:11.847641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.848370 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.848376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:11.849282 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:11.849291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.849924 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.849928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:11.850054 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:11.850071 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 10:54:11.850158 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 10:54:11.850236 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":39,"thread_start_us":125,"threads_started":1}
I20260927 10:54:11.850433 25341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:11.846661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:38211 (local address 127.14.68.1:49217)
0927 10:54:11.847027 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:11.847038 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:11.847170 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:11.847439 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:11.847442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:11.847457 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:11.847590 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:11.847595 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.848343 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.848349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:11.849275 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:11.849284 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.849917 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.849921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:11.850015 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:11.850042 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:54:11.850139 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 10:54:11.850218 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":53,"thread_start_us":92,"threads_started":1}
I20260927 10:54:11.851058 25135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8428375925249f6b7e16112190c1909" candidate_uuid: "fd9c1d91649d4b5d9225b49a3ba8940c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccd30d31bfc34101aaa48a6c93837e20" is_pre_election: true
I20260927 10:54:11.851137 25269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8428375925249f6b7e16112190c1909" candidate_uuid: "fd9c1d91649d4b5d9225b49a3ba8940c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6fe48d72aba419e91f1811af8b32ea2" is_pre_election: true
I20260927 10:54:11.851393 25135 raft_consensus.cc:2454] T f8428375925249f6b7e16112190c1909 P ccd30d31bfc34101aaa48a6c93837e20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd9c1d91649d4b5d9225b49a3ba8940c in term 0.
I20260927 10:54:11.851447 25269 raft_consensus.cc:2454] T f8428375925249f6b7e16112190c1909 P a6fe48d72aba419e91f1811af8b32ea2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd9c1d91649d4b5d9225b49a3ba8940c in term 0.
I20260927 10:54:11.851871 24936 leader_election.cc:302] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccd30d31bfc34101aaa48a6c93837e20, fd9c1d91649d4b5d9225b49a3ba8940c; no voters:
I20260927 10:54:11.852190 25340 raft_consensus.cc:2781] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:54:11.852273 25340 raft_consensus.cc:482] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:54:11.852324 25340 raft_consensus.cc:3037] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:11.853329 25340 raft_consensus.cc:504] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd30d31bfc34101aaa48a6c93837e20" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 35977 } } peers { permanent_uuid: "a6fe48d72aba419e91f1811af8b32ea2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 38211 } } peers { permanent_uuid: "fd9c1d91649d4b5d9225b49a3ba8940c" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 42585 } }
I20260927 10:54:11.853708 25340 leader_election.cc:288] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c [CANDIDATE]: Term 1 election: Requested vote from peers ccd30d31bfc34101aaa48a6c93837e20 (127.14.68.2:35977), a6fe48d72aba419e91f1811af8b32ea2 (127.14.68.3:38211)
I20260927 10:54:11.854024 25135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8428375925249f6b7e16112190c1909" candidate_uuid: "fd9c1d91649d4b5d9225b49a3ba8940c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccd30d31bfc34101aaa48a6c93837e20"
I20260927 10:54:11.854162 25135 raft_consensus.cc:3037] T f8428375925249f6b7e16112190c1909 P ccd30d31bfc34101aaa48a6c93837e20 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:11.854369 25269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8428375925249f6b7e16112190c1909" candidate_uuid: "fd9c1d91649d4b5d9225b49a3ba8940c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6fe48d72aba419e91f1811af8b32ea2"
I20260927 10:54:11.854473 25269 raft_consensus.cc:3037] T f8428375925249f6b7e16112190c1909 P a6fe48d72aba419e91f1811af8b32ea2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:11.855130 25135 raft_consensus.cc:2454] T f8428375925249f6b7e16112190c1909 P ccd30d31bfc34101aaa48a6c93837e20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd9c1d91649d4b5d9225b49a3ba8940c in term 1.
I20260927 10:54:11.855400 24936 leader_election.cc:302] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccd30d31bfc34101aaa48a6c93837e20, fd9c1d91649d4b5d9225b49a3ba8940c; no voters:
I20260927 10:54:11.855536 25269 raft_consensus.cc:2454] T f8428375925249f6b7e16112190c1909 P a6fe48d72aba419e91f1811af8b32ea2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd9c1d91649d4b5d9225b49a3ba8940c in term 1.
I20260927 10:54:11.855549 25340 raft_consensus.cc:2781] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:11.855784 25340 raft_consensus.cc:686] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c [term 1 LEADER]: Becoming Leader. State: Replica: fd9c1d91649d4b5d9225b49a3ba8940c, State: Running, Role: LEADER
I20260927 10:54:11.856108 25340 consensus_queue.cc:237] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd30d31bfc34101aaa48a6c93837e20" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 35977 } } peers { permanent_uuid: "a6fe48d72aba419e91f1811af8b32ea2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 38211 } } peers { permanent_uuid: "fd9c1d91649d4b5d9225b49a3ba8940c" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 42585 } }
I20260927 10:54:11.859081 24855 catalog_manager.cc:5495] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c reported cstate change: term changed from 0 to 1, leader changed from <none> to fd9c1d91649d4b5d9225b49a3ba8940c (127.14.68.1). New cstate: current_term: 1 leader_uuid: "fd9c1d91649d4b5d9225b49a3ba8940c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd30d31bfc34101aaa48a6c93837e20" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 35977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6fe48d72aba419e91f1811af8b32ea2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 38211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd9c1d91649d4b5d9225b49a3ba8940c" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 42585 } health_report { overall_health: HEALTHY } } }
I20260927 10:54:11.892117 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:11.888544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45328 (local address 127.14.68.1:42585)
0927 10:54:11.888698 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:11.888704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:11.888722 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:54:11.888815 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:11.888817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:11.888869 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:11.888983 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:11.889263 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.889768 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.890645 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:11.890863 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:11.890973 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:11.891292 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:11.891740 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:11.891878 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:11.891918 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 10:54:11.891973 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 10:54:11.897804 25269 raft_consensus.cc:1261] T f8428375925249f6b7e16112190c1909 P a6fe48d72aba419e91f1811af8b32ea2 [term 1 FOLLOWER]: Refusing update from remote peer fd9c1d91649d4b5d9225b49a3ba8940c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:11.897804 25135 raft_consensus.cc:1261] T f8428375925249f6b7e16112190c1909 P ccd30d31bfc34101aaa48a6c93837e20 [term 1 FOLLOWER]: Refusing update from remote peer fd9c1d91649d4b5d9225b49a3ba8940c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:11.898492 25340 consensus_queue.cc:1035] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6fe48d72aba419e91f1811af8b32ea2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 38211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:11.898691 25343 consensus_queue.cc:1035] T f8428375925249f6b7e16112190c1909 P fd9c1d91649d4b5d9225b49a3ba8940c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccd30d31bfc34101aaa48a6c93837e20" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 35977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:11.905905 25346 mvcc.cc:205] Tried to move back new op lower bound from 7333914426966392832 to 7333914426805305344. Current Snapshot: MvccSnapshot[applied={T|T < 7333914426966392832}]
I20260927 10:54:11.915230 24856 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:54:11.915380 24856 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:54:11.917680 24856 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 881220d313824adaa13abcbd415014c5: Sending DeleteTablet for 3 replicas of tablet f8428375925249f6b7e16112190c1909
I20260927 10:54:11.918653 25115 tablet_service.cc:1514] Processing DeleteTablet for tablet f8428375925249f6b7e16112190c1909 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:11 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:57936
I20260927 10:54:11.918711 25249 tablet_service.cc:1514] Processing DeleteTablet for tablet f8428375925249f6b7e16112190c1909 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:11 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:49020
I20260927 10:54:11.918777 24981 tablet_service.cc:1514] Processing DeleteTablet for tablet f8428375925249f6b7e16112190c1909 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:11 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:45322
I20260927 10:54:11.919188 25357 tablet_replica.cc:330] T f8428375925249f6b7e16112190c1909 P a6fe48d72aba419e91f1811af8b32ea2: stopping tablet replica
I20260927 10:54:11.919520 25357 raft_consensus.cc:2229] T f8428375925249f6b7e16112190c1909 P a6fe48d72aba419e91f1811af8b32ea2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:54:11.919780 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 24918
I20260927 10:54:11.920771 25357 raft_consensus.cc:2258] T f8428375925249f6b7e16112190c1909 P a6fe48d72aba419e91f1811af8b32ea2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:11.919685 25355 tablet_replica.cc:330] T f8428375925249f6b7e16112190c1909 P ccd30d31bfc34101aaa48a6c93837e20: stopping tablet replica
I20260927 10:54:11.921600 25355 raft_consensus.cc:2229] T f8428375925249f6b7e16112190c1909 P ccd30d31bfc34101aaa48a6c93837e20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:54:11.921906 25355 raft_consensus.cc:2258] T f8428375925249f6b7e16112190c1909 P ccd30d31bfc34101aaa48a6c93837e20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:11.922829 25357 ts_tablet_manager.cc:1905] T f8428375925249f6b7e16112190c1909 P a6fe48d72aba419e91f1811af8b32ea2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:54:11.923189 25355 ts_tablet_manager.cc:1905] T f8428375925249f6b7e16112190c1909 P ccd30d31bfc34101aaa48a6c93837e20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:54:11.925572 25355 ts_tablet_manager.cc:1918] T f8428375925249f6b7e16112190c1909 P ccd30d31bfc34101aaa48a6c93837e20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:54:11.925685 25355 log.cc:1196] T f8428375925249f6b7e16112190c1909 P ccd30d31bfc34101aaa48a6c93837e20: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/f8428375925249f6b7e16112190c1909
I20260927 10:54:11.926021 25355 ts_tablet_manager.cc:1939] T f8428375925249f6b7e16112190c1909 P ccd30d31bfc34101aaa48a6c93837e20: Deleting consensus metadata
I20260927 10:54:11.926236 25357 ts_tablet_manager.cc:1918] T f8428375925249f6b7e16112190c1909 P a6fe48d72aba419e91f1811af8b32ea2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:54:11.926334 25357 log.cc:1196] T f8428375925249f6b7e16112190c1909 P a6fe48d72aba419e91f1811af8b32ea2: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/f8428375925249f6b7e16112190c1909
I20260927 10:54:11.926723 25357 ts_tablet_manager.cc:1939] T f8428375925249f6b7e16112190c1909 P a6fe48d72aba419e91f1811af8b32ea2: Deleting consensus metadata
I20260927 10:54:11.927205 24842 catalog_manager.cc:4841] TS ccd30d31bfc34101aaa48a6c93837e20 (127.14.68.2:35977): tablet f8428375925249f6b7e16112190c1909 (table test-table [id=9141848e1b944812b8dc5795b8a0f2f1]) successfully deleted
W20260927 10:54:11.927978 24843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.68.1:42585 (error 108)
I20260927 10:54:11.928488 24843 catalog_manager.cc:4841] TS a6fe48d72aba419e91f1811af8b32ea2 (127.14.68.3:38211): tablet f8428375925249f6b7e16112190c1909 (table test-table [id=9141848e1b944812b8dc5795b8a0f2f1]) successfully deleted
I20260927 10:54:11.928678 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 25052
I20260927 10:54:11.929059 25329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:11.928740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:42585 (local address 127.0.0.1:45334)
0927 10:54:11.928908 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:11.928973 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.1:42585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260927 10:54:11.929776 24843 catalog_manager.cc:4584] TS fd9c1d91649d4b5d9225b49a3ba8940c (127.14.68.1:42585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f8428375925249f6b7e16112190c1909: Network error: Client connection negotiation failed: client connection to 127.14.68.1:42585: connect: Connection refused (error 111)
I20260927 10:54:11.934046 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 25186
I20260927 10:54:11.939406 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 24824
2026-09-27T10:54:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2285 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:11 dist-test-slave-tz51 krb5kdc[25361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:11 dist-test-slave-tz51 krb5kdc[25361](info): set up 2 sockets
Sep 27 10:54:11 dist-test-slave-tz51 krb5kdc[25361](info): commencing operation
krb5kdc: starting...
W20260927 10:54:12.980787 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:54:13 dist-test-slave-tz51 krb5kdc[25361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.68.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.68.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.68.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/rangeradmin_127.14.68.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.68.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/rangeradmin_127.14.68.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.68.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.68.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.68.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/rangerlookup_127.14.68.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.68.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/rangerlookup_127.14.68.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.68.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.68.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.68.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/HTTP_127.14.68.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.68.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/HTTP_127.14.68.20@KRBTEST.COM.keytab.
I20260927 10:54:13.151115 14608 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskAZZUsq/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/postgres -l logfile start
2026-09-27 10:54:15.636 UTC [25391] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 10:54:15.636 UTC [25391] LOG: listening on IPv4 address "127.14.68.20", port 40037
2026-09-27 10:54:15.639 UTC [25391] LOG: listening on Unix socket "/tmp/.s.PGSQL.40037"
2026-09-27 10:54:15.661 UTC [25394] LOG: database system was shut down at 2026-09-27 10:54:14 UTC
2026-09-27 10:54:15.665 UTC [25391] LOG: database system is ready to accept connections
I20260927 10:54:16.612423 14608 mini_postgres.cc:96] Postgres bound to 40037
2026-09-27 10:54:16.618 UTC [25402] FATAL: database "slave" does not exist
127.14.68.20:40037 - accepting connections
I20260927 10:54:16.619240 14608 mini_ranger.cc:161] Starting Ranger...
I20260927 10:54:16.633869 14608 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 10:54:17.123445 14608 mini_ranger.cc:87] Created ranger Postgres database
I20260927 10:54:17.123569 14608 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ranger-admin
2026-09-27 10:54:17,430 [I] DB FLAVOR :POSTGRES
2026-09-27 10:54:17,431 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 10:54:17,431 [I] Checking connection..
2026-09-27 10:54:17,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:40037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 10:54:17,656 [I] Checking connection passed.
2026-09-27 10:54:17,657 [I] --------- Verifying version history table ---------
2026-09-27 10:54:17,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:40037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:54:17,916 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 10:54:17,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:40037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:54:18,175 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 10:54:18,175 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 10:54:18,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:40037/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 10:54:18.396 UTC [25466] WARNING: there is no transaction in progress
2026-09-27 10:54:18,402 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 10:54:18,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:40037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:54:18,640 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 10:54:18,640 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 10:54:18,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:40037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 10:54:18,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:40037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 10:54:19,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:40037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tz51.c.gcp-upstream.internal','N') ;"
2026-09-27 10:54:19,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:40037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 10:54:19,618 [I] Table x_portal_user does not exist in database ranger
2026-09-27 10:54:19,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:40037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 10:54:19,885 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 10:54:19,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:40037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 10:54:20,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:40037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 10:54:20,368 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 10:54:20,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:40037/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 10:54:21.340 UTC [25574] WARNING: there is no transaction in progress
2026-09-27 10:54:21.356 UTC [25574] WARNING: there is no transaction in progress
2026-09-27 10:54:21.373 UTC [25574] WARNING: there is no transaction in progress
2026-09-27 10:54:21.389 UTC [25574] WARNING: there is no transaction in progress
2026-09-27 10:54:21.405 UTC [25574] WARNING: there is no transaction in progress
2026-09-27 10:54:21.425 UTC [25574] WARNING: there is no transaction in progress
2026-09-27 10:54:21.490 UTC [25574] WARNING: there is no transaction in progress
2026-09-27 10:54:21.497 UTC [25574] WARNING: there is no transaction in progress
2026-09-27 10:54:21.511 UTC [25574] WARNING: there is no transaction in progress
2026-09-27 10:54:21.521 UTC [25574] WARNING: there is no transaction in progress
2026-09-27 10:54:21.531 UTC [25574] WARNING: there is no transaction in progress
2026-09-27 10:54:21.546 UTC [25574] WARNING: there is no transaction in progress
2026-09-27 10:54:21.555 UTC [25574] WARNING: there is no transaction in progress
2026-09-27 10:54:21.561 UTC [25574] WARNING: there is no transaction in progress
2026-09-27 10:54:21.568 UTC [25574] WARNING: there is no transaction in progress
2026-09-27 10:54:21.575 UTC [25574] WARNING: there is no transaction in progress
2026-09-27 10:54:22,237 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 10:54:22,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:40037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 10:54:22,459 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 10:54:22,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:40037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 10:54:22,721 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 10:54:22,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:40037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 10:54:22,956 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 10:54:22,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:40037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 10:54:23,209 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:40037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 10:54:23,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:40037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tz51.c.gcp-upstream.internal';"
2026-09-27 10:54:23,682 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 10:54:23,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:40037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 10:54:23,916 [I] DB_PATCHES have already been applied
I20260927 10:54:23.921829 14608 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ranger-admin:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskAZZUsq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 10:54:23.921909 14608 mini_ranger.cc:193] Using host: 127.14.68.20
I20260927 10:54:23.925354 14608 mini_ranger.cc:227] Ranger admin URL: http://127.14.68.20:45533
Sep 27, 2026 10:54:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:54:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:54:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:54:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:54:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:54:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:54:24 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 10:54:25.036373 14608 mini_ranger.cc:160] Time spent starting Ranger: real 8.417s user 0.002s sys 0.008s
Sep 27, 2026 10:54:45 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:54:45 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 27, 2026 10:54:45 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:54:45 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:54:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:54:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:54:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:54:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:54:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:54:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:54:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:54:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:54:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:54:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:54:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:54:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:54:45 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:54:46 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 10:54:47.020746 14608 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.68.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.68.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.68.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/rangerkms_127.14.68.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.68.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/rangerkms_127.14.68.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.68.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.68.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.68.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/HTTP_127.14.68.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.68.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/HTTP_127.14.68.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 10:54:47 dist-test-slave-tz51 krb5kdc[25361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506487, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 10:54:47.127154 14608 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 10:54:47.137215 14608 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 10:54:48.001375 14608 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 10:54:48.001477 14608 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ranger-kms
I20260927 10:54:48.001578 14608 mini_ranger_kms.cc:227] Using postgres at 127.14.68.20:40037
2026-09-27 10:54:48,174 [I] DB FLAVOR :POSTGRES
2026-09-27 10:54:48,174 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 10:54:48,174 [I] Checking connection
2026-09-27 10:54:48,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:40037/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 10:54:48,401 [I] connection success
2026-09-27 10:54:48,402 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 10:54:48,402 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 10:54:48,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:40037/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 10:54:48,642 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 10:54:48,642 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 10:54:48,642 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 10:54:48,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:40037/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 10:54:48,917 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 10:54:49.337664 14608 mini_ranger_kms.cc:310] Created kms service
I20260927 10:54:49.539811 14608 mini_ranger_kms.cc:326] Created kudu user
I20260927 10:54:49.583051 14608 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 10:54:49 dist-test-slave-tz51 krb5kdc[25361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506487, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.68.20@KRBTEST.COM
I20260927 10:54:49.995371 14608 mini_ranger_kms.cc:384] Added ranger policy
I20260927 10:54:49.995533 14608 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ranger-kms:/tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 10:54:49.995563 14608 mini_ranger_kms.cc:240] Using host: 127.14.68.20
I20260927 10:54:49.999825 14608 mini_ranger_kms.cc:276] Ranger KMS PID: 25754
I20260927 10:54:49.999951 14608 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.68.20:35425
Sep 27, 2026 10:54:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:54:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:54:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:54:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:54:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:54:51 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 10:54:52.226105 14608 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.099s user 0.010s sys 0.146s
I20260927 10:54:52.226239 14608 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 10:54:52.226353 14608 mini_ranger_kms.cc:401] 127.14.68.20:35425/kms/v1/keys
Sep 27 10:54:52 dist-test-slave-tz51 krb5kdc[25361](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790506492, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.68.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 10:54:52 dist-test-slave-tz51 krb5kdc[25361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506492, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.68.20@KRBTEST.COM for HTTP/127.14.68.20@KRBTEST.COM
Sep 27 10:54:53 dist-test-slave-tz51 krb5kdc[25361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506492, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.68.20@KRBTEST.COM for HTTP/127.14.68.20@KRBTEST.COM
Sep 27 10:54:55 dist-test-slave-tz51 krb5kdc[25361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:54:55.334900 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:43929
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:37443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:43929
--ranger_config_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.68.20:35425/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:55.442878 25805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:55.443138 25805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:55.443185 25805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:55.447175 25805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:55.447234 25805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:55.447254 25805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:55.447271 25805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:55.447286 25805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:55.452828 25805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:37443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.68.20:35425/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:43929
--ranger_config_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:43929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:54:55.453884 25805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:55.454921 25805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:55.464018 25811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:55.464015 25813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:55.464012 25810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:55.464577 25805 server_base.cc:1034] running on GCE node
I20260927 10:54:55.465233 25805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:55.466615 25805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:55.467787 25805 hybrid_clock.cc:648] HybridClock initialized: now 1790506495467764 us; error 32 us; skew 500 ppm
Sep 27 10:54:55 dist-test-slave-tz51 krb5kdc[25361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506495, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:55.475339 25805 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:54:55.476354 25805 webserver.cc:460] Webserver started at http://127.14.68.62:41787/ using document root <none> and password file <none>
I20260927 10:54:55.476924 25805 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:55.477005 25805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:55.477211 25805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:54:55 dist-test-slave-tz51 krb5kdc[25361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506495, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for HTTP/127.14.68.20@KRBTEST.COM
I20260927 10:54:55.536077 25805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "a0971230edc248058a4eeeee7451620c"
format_stamp: "Formatted at 2026-09-27 10:54:55 on dist-test-slave-tz51"
server_key: "15aaa105518fbb7fb7d8ea12698f7cf6"
server_key_iv: "56a5b519447b01da100efc8e68cb5c9c"
server_key_version: "kuduclusterkey@0"
I20260927 10:54:55.536726 25805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "a0971230edc248058a4eeeee7451620c"
format_stamp: "Formatted at 2026-09-27 10:54:55 on dist-test-slave-tz51"
server_key: "15aaa105518fbb7fb7d8ea12698f7cf6"
server_key_iv: "56a5b519447b01da100efc8e68cb5c9c"
server_key_version: "kuduclusterkey@0"
I20260927 10:54:55.540728 25805 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:54:55.557639 25820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:55.559020 25805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:54:55.559123 25805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "a0971230edc248058a4eeeee7451620c"
format_stamp: "Formatted at 2026-09-27 10:54:55 on dist-test-slave-tz51"
server_key: "15aaa105518fbb7fb7d8ea12698f7cf6"
server_key_iv: "56a5b519447b01da100efc8e68cb5c9c"
server_key_version: "kuduclusterkey@0"
I20260927 10:54:55.559275 25805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:55.579373 25805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:55.582458 25805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:55.582664 25805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:55.590588 25805 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:43929
I20260927 10:54:55.590648 25872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:43929 every 8 connection(s)
I20260927 10:54:55.591558 25805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:55.591945 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 25805
I20260927 10:54:55.592087 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:54:55.592409 14608 external_mini_cluster.cc:1428] Setting key 3f808b2f7ba591559df2c03843a556dc
I20260927 10:54:55.594854 25873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:55.600955 25873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c: Bootstrap starting.
Sep 27 10:54:55 dist-test-slave-tz51 krb5kdc[25361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:54:55.603525 25873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:55.604463 25873 log.cc:824] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:55.606782 25873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c: No bootstrap required, opened a new log
I20260927 10:54:55.609776 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:55.593809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38726 (local address 127.14.68.62:43929)
0927 10:54:55.594272 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:55.594292 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:55.594336 (+ 44us) server_negotiation.cc:407] Connection header received
0927 10:54:55.595243 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:55.595268 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:55.595730 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:55.596099 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:55.596954 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:55.598536 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:55.599829 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:55.600577 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:55.600759 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:55.603942 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:55.603948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:55.603963 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:55.603992 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:55.606555 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:55.607079 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:55.607085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:55.607091 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:55.607231 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:55.607550 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:55.607553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:55.607555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:55.608071 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:55.608255 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:55.608546 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 10:54:55.608827 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":143,"threads_started":1}
I20260927 10:54:55.610814 25873 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0971230edc248058a4eeeee7451620c" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43929 } }
I20260927 10:54:55.611095 25873 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:55.611193 25873 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0971230edc248058a4eeeee7451620c, State: Initialized, Role: FOLLOWER
I20260927 10:54:55.611668 25873 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0971230edc248058a4eeeee7451620c" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43929 } }
I20260927 10:54:55.611868 25873 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:55.611949 25873 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:55.612087 25873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:55.613022 25873 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0971230edc248058a4eeeee7451620c" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43929 } }
I20260927 10:54:55.613364 25873 leader_election.cc:302] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0971230edc248058a4eeeee7451620c; no voters:
I20260927 10:54:55.613692 25873 leader_election.cc:288] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:55.613772 25878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:55.613987 25878 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c [term 1 LEADER]: Becoming Leader. State: Replica: a0971230edc248058a4eeeee7451620c, State: Running, Role: LEADER
I20260927 10:54:55.614326 25878 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0971230edc248058a4eeeee7451620c" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43929 } }
I20260927 10:54:55.614980 25873 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:55.615442 25878 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0971230edc248058a4eeeee7451620c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0971230edc248058a4eeeee7451620c" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43929 } } }
I20260927 10:54:55.615624 25878 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:55.615907 25879 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0971230edc248058a4eeeee7451620c. Latest consensus state: current_term: 1 leader_uuid: "a0971230edc248058a4eeeee7451620c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0971230edc248058a4eeeee7451620c" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43929 } } }
I20260927 10:54:55.616009 25879 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:55.616132 25884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:55.619364 25884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:55.619781 25873 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 10:54:55.628996 25884 catalog_manager.cc:1347] Generated new cluster ID: 4c5155200a00466485d0de22e0f217e4
I20260927 10:54:55.629122 25884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:55.645133 25884 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:55.646471 25884 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:55.661965 25884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c: Generated new TSK 0
I20260927 10:54:55.662932 25884 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
I20260927 10:54:56.323114 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.68.20:35425/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:43929
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:37443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:56.467814 25918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:56.468055 25918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:56.468130 25918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:56.472108 25918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:56.472182 25918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:56.472270 25918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:54:56.477494 25918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:37443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.68.20:35425/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:43929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:54:56.478525 25918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:56.479569 25918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:56.490141 25926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:56.490145 25923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:56.490399 25924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:56.491501 25918 server_base.cc:1034] running on GCE node
I20260927 10:54:56.491973 25918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:56.492705 25918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:56.493893 25918 hybrid_clock.cc:648] HybridClock initialized: now 1790506496493856 us; error 48 us; skew 500 ppm
Sep 27 10:54:56 dist-test-slave-tz51 krb5kdc[25361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506496, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:56.497498 25918 init.cc:377] Logged in from keytab as kudu/127.14.68.1@KRBTEST.COM (short username kudu)
I20260927 10:54:56.498611 25918 webserver.cc:460] Webserver started at http://127.14.68.1:35655/ using document root <none> and password file <none>
I20260927 10:54:56.499320 25918 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:56.499419 25918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:56.499689 25918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:54:56 dist-test-slave-tz51 krb5kdc[25361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506496, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for HTTP/127.14.68.20@KRBTEST.COM
I20260927 10:54:56.519052 25918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "df5634aa9bb141518ab3877ee74eafec"
format_stamp: "Formatted at 2026-09-27 10:54:56 on dist-test-slave-tz51"
server_key: "1e140f1c1a5761f1743ead87a96972c9"
server_key_iv: "503d673fe4e0d567cf3f9d0d3b3ec1e1"
server_key_version: "kuduclusterkey@0"
I20260927 10:54:56.520687 25918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "df5634aa9bb141518ab3877ee74eafec"
format_stamp: "Formatted at 2026-09-27 10:54:56 on dist-test-slave-tz51"
server_key: "1e140f1c1a5761f1743ead87a96972c9"
server_key_iv: "503d673fe4e0d567cf3f9d0d3b3ec1e1"
server_key_version: "kuduclusterkey@0"
I20260927 10:54:56.529016 25918 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260927 10:54:56.544215 25935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:56.545548 25918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:54:56.545682 25918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "df5634aa9bb141518ab3877ee74eafec"
format_stamp: "Formatted at 2026-09-27 10:54:56 on dist-test-slave-tz51"
server_key: "1e140f1c1a5761f1743ead87a96972c9"
server_key_iv: "503d673fe4e0d567cf3f9d0d3b3ec1e1"
server_key_version: "kuduclusterkey@0"
I20260927 10:54:56.545818 25918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:56.578362 25918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:56.581991 25918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:56.582231 25918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:56.582926 25918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:56.584003 25918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:56.584115 25918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:56.584250 25918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:56.584334 25918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:56.595659 25918 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:42793
I20260927 10:54:56.596760 25918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:54:56.598711 26048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:42793 every 8 connection(s)
I20260927 10:54:56.599947 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 25918
I20260927 10:54:56.600081 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:54:56.600430 14608 external_mini_cluster.cc:1428] Setting key 343e2536307d4bdb5e1487ad834358e3
WARNING: no policy specified for kudu/127.14.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.2@KRBTEST.COM" created.
Sep 27 10:54:56 dist-test-slave-tz51 krb5kdc[25361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506496, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:54:56.625028 26053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:56.604723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:43929 (local address 127.14.68.1:57281)
0927 10:54:56.605236 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:56.605286 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:56.606271 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:56.612102 (+ 5831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:56.612112 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:56.612496 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:56.613201 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:56.613224 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.614699 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.614703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:56.616279 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:56.616289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.616947 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.616951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:56.617144 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:56.617829 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:56.617849 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:56.619546 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:56.621844 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:56.621850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:56.621865 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:56.622188 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:56.622504 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:56.622507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:56.622509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:56.622693 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:56.623174 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:56.623180 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:56.623487 (+ 307us) client_negotiation.cc:769] Sending connection context
0927 10:54:56.623673 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 10:54:56.624008 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":145,"threads_started":1}
I20260927 10:54:56.625813 26052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:56.604187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:57281 (local address 127.14.68.62:43929)
0927 10:54:56.611641 (+ 7454us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:56.611650 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:56.611666 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:54:56.611734 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:56.611737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:56.611785 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:56.613401 (+ 1616us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:56.613561 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.614555 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.616421 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.616772 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.619737 (+ 2965us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:56.619865 (+ 128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:56.619869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:56.619871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:56.619888 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:56.621719 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:56.622313 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:56.622316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:56.622318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:56.622401 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:56.622816 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:56.622819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:56.622821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:56.623037 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:56.625430 (+ 2393us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:56.625514 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 10:54:56.625637 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7374,"thread_start_us":82,"threads_started":1}
I20260927 10:54:56.626847 26049 heartbeater.cc:347] Connected to a master server at 127.14.68.62:43929
I20260927 10:54:56.627087 26049 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:56.627779 26049 heartbeater.cc:511] Master 127.14.68.62:43929 requested a full tablet report, sending...
I20260927 10:54:56.629595 25837 ts_manager.cc:194] Registered new tserver with Master: df5634aa9bb141518ab3877ee74eafec (127.14.68.1:42793)
I20260927 10:54:56.631102 25837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:57281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
I20260927 10:54:56.643371 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.68.20:35425/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:43929
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:37443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:54:56 dist-test-slave-tz51 krb5kdc[25361](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790506496, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 10:54:56.769619 26057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:56.769938 26057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:56.770053 26057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:56.774580 26057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:56.774665 26057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:56.774803 26057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:54:56.785414 26057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:37443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.68.20:35425/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:43929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:54:56.786624 26057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:56.787647 26057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:56.797031 26066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:56.797029 26064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:56.797535 26063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:56.798245 26057 server_base.cc:1034] running on GCE node
I20260927 10:54:56.798657 26057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:56.799326 26057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:56.800506 26057 hybrid_clock.cc:648] HybridClock initialized: now 1790506496800435 us; error 87 us; skew 500 ppm
Sep 27 10:54:56 dist-test-slave-tz51 krb5kdc[25361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506496, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:56.804133 26057 init.cc:377] Logged in from keytab as kudu/127.14.68.2@KRBTEST.COM (short username kudu)
I20260927 10:54:56.805222 26057 webserver.cc:460] Webserver started at http://127.14.68.2:42195/ using document root <none> and password file <none>
I20260927 10:54:56.805914 26057 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:56.806000 26057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:56.806232 26057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:54:56 dist-test-slave-tz51 krb5kdc[25361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506496, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for HTTP/127.14.68.20@KRBTEST.COM
I20260927 10:54:56.829281 26057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "5c459bdd257f4640981d97f0880921f7"
format_stamp: "Formatted at 2026-09-27 10:54:56 on dist-test-slave-tz51"
server_key: "f190c9f477537f1dca55b2e38c1d0aca"
server_key_iv: "1d10ceb8a828f0b236bc32550d4e1c5b"
server_key_version: "kuduclusterkey@0"
I20260927 10:54:56.830832 26057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "5c459bdd257f4640981d97f0880921f7"
format_stamp: "Formatted at 2026-09-27 10:54:56 on dist-test-slave-tz51"
server_key: "f190c9f477537f1dca55b2e38c1d0aca"
server_key_iv: "1d10ceb8a828f0b236bc32550d4e1c5b"
server_key_version: "kuduclusterkey@0"
I20260927 10:54:56.839534 26057 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260927 10:54:56.871551 26073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:56.873234 26057 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.001s
I20260927 10:54:56.873368 26057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "5c459bdd257f4640981d97f0880921f7"
format_stamp: "Formatted at 2026-09-27 10:54:56 on dist-test-slave-tz51"
server_key: "f190c9f477537f1dca55b2e38c1d0aca"
server_key_iv: "1d10ceb8a828f0b236bc32550d4e1c5b"
server_key_version: "kuduclusterkey@0"
I20260927 10:54:56.873515 26057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:56.912094 26057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:56.915735 26057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:56.915920 26057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:56.916651 26057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:56.917711 26057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:56.917778 26057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:56.917831 26057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:56.917863 26057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:56.929548 26057 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:35085
I20260927 10:54:56.929800 26186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:35085 every 8 connection(s)
I20260927 10:54:56.930760 26057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:54:56.932258 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 26057
I20260927 10:54:56.932401 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:54:56.932718 14608 external_mini_cluster.cc:1428] Setting key dbbae3de5d795537e07f98c9a63720e0
Sep 27 10:54:56 dist-test-slave-tz51 krb5kdc[25361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506496, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.68.3@KRBTEST.COM; defaulting to no policy
I20260927 10:54:56.949585 26052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:56.934348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:48399 (local address 127.14.68.62:43929)
0927 10:54:56.934481 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:56.934484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:56.935873 (+ 1389us) server_negotiation.cc:407] Connection header received
0927 10:54:56.937282 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:56.937287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:56.937347 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:56.937405 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:56.938643 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.939529 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.941468 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.941919 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.942017 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:56.945288 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:56.945295 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:56.945301 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:56.945336 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:56.947125 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:56.947764 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:56.947768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:56.947769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:56.947853 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:56.948238 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:56.948241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:56.948243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:56.948457 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:56.948537 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:56.949279 (+ 742us) server_negotiation.cc:299] Negotiation successful
0927 10:54:56.949413 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 10:54:56.950582 26189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:56.933346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:43929 (local address 127.14.68.2:48399)
0927 10:54:56.935718 (+ 2372us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:56.935777 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:56.937003 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:56.937761 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:56.937768 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:56.938027 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:56.938501 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:56.938513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.939715 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.939722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:56.941279 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:56.941293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.942243 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.942250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:56.942468 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:56.943239 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:56.943264 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:56.945084 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:56.947272 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:56.947278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:56.947293 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:56.947613 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:56.947959 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:56.947962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:56.947964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:56.948127 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:56.948601 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:56.948607 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:56.948919 (+ 312us) client_negotiation.cc:769] Sending connection context
0927 10:54:56.949175 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 10:54:56.949488 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2099,"thread_start_us":147,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.3@KRBTEST.COM" created.
I20260927 10:54:56.951941 26187 heartbeater.cc:347] Connected to a master server at 127.14.68.62:43929
I20260927 10:54:56.952199 26187 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:56.952754 26187 heartbeater.cc:511] Master 127.14.68.62:43929 requested a full tablet report, sending...
I20260927 10:54:56.953807 25837 ts_manager.cc:194] Registered new tserver with Master: 5c459bdd257f4640981d97f0880921f7 (127.14.68.2:35085)
I20260927 10:54:56.954540 25837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:48399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
I20260927 10:54:56.973021 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.68.20:35425/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:43929
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:37443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:57.131352 26192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:57.131810 26192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:57.131948 26192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:57.137925 26192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:57.138082 26192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:57.138240 26192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:54:57.146166 26192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:37443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.68.20:35425/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:43929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:54:57.147821 26192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:57.149232 26192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 27 10:54:57 dist-test-slave-tz51 krb5kdc[25361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506496, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for HTTP/127.14.68.20@KRBTEST.COM
W20260927 10:54:57.159802 26200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:57.160274 26197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:57.161067 26198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:57.161293 26192 server_base.cc:1034] running on GCE node
I20260927 10:54:57.161908 26192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:57.162679 26192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:57.166492 26192 hybrid_clock.cc:648] HybridClock initialized: now 1790506497166419 us; error 82 us; skew 500 ppm
Sep 27 10:54:57 dist-test-slave-tz51 krb5kdc[25361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506497, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:57.174206 26192 init.cc:377] Logged in from keytab as kudu/127.14.68.3@KRBTEST.COM (short username kudu)
I20260927 10:54:57.175422 26192 webserver.cc:460] Webserver started at http://127.14.68.3:41003/ using document root <none> and password file <none>
I20260927 10:54:57.176232 26192 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:57.176368 26192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:57.176743 26192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:54:57 dist-test-slave-tz51 krb5kdc[25361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506497, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for HTTP/127.14.68.20@KRBTEST.COM
I20260927 10:54:57.193522 26192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "5ec74237395f459692f6053170bad74f"
format_stamp: "Formatted at 2026-09-27 10:54:57 on dist-test-slave-tz51"
server_key: "61d0a79774f7e217e377f602551e95d3"
server_key_iv: "b458268f01208ee3872d0de00733ca60"
server_key_version: "kuduclusterkey@0"
I20260927 10:54:57.194290 26192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "5ec74237395f459692f6053170bad74f"
format_stamp: "Formatted at 2026-09-27 10:54:57 on dist-test-slave-tz51"
server_key: "61d0a79774f7e217e377f602551e95d3"
server_key_iv: "b458268f01208ee3872d0de00733ca60"
server_key_version: "kuduclusterkey@0"
I20260927 10:54:57.206291 26192 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.004s sys 0.000s
I20260927 10:54:57.222699 26207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:57.224331 26192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:54:57.224459 26192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "5ec74237395f459692f6053170bad74f"
format_stamp: "Formatted at 2026-09-27 10:54:57 on dist-test-slave-tz51"
server_key: "61d0a79774f7e217e377f602551e95d3"
server_key_iv: "b458268f01208ee3872d0de00733ca60"
server_key_version: "kuduclusterkey@0"
I20260927 10:54:57.224653 26192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:57.247555 26192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:57.251246 26192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:57.251470 26192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:57.252202 26192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:57.253197 26192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:57.253258 26192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:57.253310 26192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:57.253338 26192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:57.263079 26192 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:45073
I20260927 10:54:57.263190 26320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:45073 every 8 connection(s)
I20260927 10:54:57.264132 26192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:54:57.265905 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 26192
I20260927 10:54:57.266028 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:54:57.266336 14608 external_mini_cluster.cc:1428] Setting key 4bfa8dbd5eddc83dc95ddc287f34bff9
Sep 27 10:54:57 dist-test-slave-tz51 krb5kdc[25361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506497, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:54:57.280423 26052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.267553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:47109 (local address 127.14.68.62:43929)
0927 10:54:57.267677 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:57.267682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:57.268797 (+ 1115us) server_negotiation.cc:407] Connection header received
0927 10:54:57.269884 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:57.269887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:57.269934 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:57.270014 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:57.271295 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.272121 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.273670 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.273885 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.273964 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:57.276516 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:57.276520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:57.276523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:57.276550 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:57.278137 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:57.278666 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:57.278672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:57.278674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:57.278786 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:57.279178 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:57.279181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:57.279183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:57.279390 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:57.279500 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:57.280144 (+ 644us) server_negotiation.cc:299] Negotiation successful
0927 10:54:57.280271 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:54:57.281292 26323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.266565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:43929 (local address 127.14.68.3:47109)
0927 10:54:57.268644 (+ 2079us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:57.268694 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:57.269669 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:57.270181 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:57.270189 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:57.270511 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:57.271140 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:57.271159 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.272249 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.272253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:57.273534 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:57.273546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.274037 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.274042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:57.274229 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:57.274836 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:57.274857 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:57.276358 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:57.278244 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:57.278249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:57.278260 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:57.278517 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:57.278897 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:57.278900 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:57.278902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:57.279088 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:57.279566 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:57.279574 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:57.279895 (+ 321us) client_negotiation.cc:769] Sending connection context
0927 10:54:57.280136 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 10:54:57.280401 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1856,"thread_start_us":123,"threads_started":1}
I20260927 10:54:57.282373 26321 heartbeater.cc:347] Connected to a master server at 127.14.68.62:43929
I20260927 10:54:57.282631 26321 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:57.283186 26321 heartbeater.cc:511] Master 127.14.68.62:43929 requested a full tablet report, sending...
I20260927 10:54:57.284272 25837 ts_manager.cc:194] Registered new tserver with Master: 5ec74237395f459692f6053170bad74f (127.14.68.3:45073)
I20260927 10:54:57.284822 25837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:47109
I20260927 10:54:57.291973 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:54:57.303663 26052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.294731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38738 (local address 127.14.68.62:43929)
0927 10:54:57.294870 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:57.294876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:57.295081 (+ 205us) server_negotiation.cc:407] Connection header received
0927 10:54:57.295219 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:57.295224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:57.295276 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:57.295365 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:57.296111 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.296953 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.297999 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.298292 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.298408 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:57.300134 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:57.300138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:57.300141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:57.300165 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:57.301884 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:57.302396 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:57.302400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:57.302402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:57.302490 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:57.302798 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:57.302802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:57.302803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:57.303013 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:57.303162 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:57.303397 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 10:54:57.303509 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:54:57.306648 25837 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:54:57.311002 25837 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:54:57.329519 26333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.321697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:45073 (local address 127.0.0.1:36156)
0927 10:54:57.322139 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:57.322190 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:57.322315 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:57.323508 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:57.323515 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:57.323541 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:57.323682 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:57.323689 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.326243 (+ 2554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.326247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:57.327684 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:57.327718 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.328934 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.328938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:57.329053 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:57.329120 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 10:54:57.329210 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 10:54:57.329354 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":112,"threads_started":1}
I20260927 10:54:57.330850 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.322572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36156 (local address 127.14.68.3:45073)
0927 10:54:57.323002 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:57.323014 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:57.323036 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:54:57.323122 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:57.323127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:57.323330 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:57.323877 (+ 547us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:57.324003 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.326105 (+ 2102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.327876 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.328795 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.330415 (+ 1620us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:57.330558 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:57.330610 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 10:54:57.330708 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":151,"threads_started":1}
I20260927 10:54:57.331919 26335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.322978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:42793 (local address 127.0.0.1:43396)
0927 10:54:57.324892 (+ 1914us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:57.324907 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:57.324994 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:57.325911 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:57.325914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:57.325932 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:57.326035 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:57.326041 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.328272 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.328275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:57.329547 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:57.329558 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.331501 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.331504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:57.331609 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:57.331638 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 10:54:57.331706 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 10:54:57.331787 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1854,"thread_start_us":114,"threads_started":1}
I20260927 10:54:57.332520 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.323402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.14.68.1:42793)
0927 10:54:57.325261 (+ 1859us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:57.325268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:57.325291 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:54:57.325364 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:57.325369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:57.325592 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:57.325751 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:57.326203 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.328139 (+ 1936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.329692 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.330531 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.330646 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:57.332233 (+ 1587us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:57.332283 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 10:54:57.332374 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1750,"thread_start_us":86,"threads_started":1}
I20260927 10:54:57.334144 26255 tablet_service.cc:1467] Processing CreateTablet for tablet a7fe3ae481ca401e9061542f130dc33c (DEFAULT_TABLE table=test-table [id=f1c757e43fa64b8cb4efb6f5742838d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:57.335235 26255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7fe3ae481ca401e9061542f130dc33c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:57.335152 25983 tablet_service.cc:1467] Processing CreateTablet for tablet a7fe3ae481ca401e9061542f130dc33c (DEFAULT_TABLE table=test-table [id=f1c757e43fa64b8cb4efb6f5742838d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:57.336239 25983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7fe3ae481ca401e9061542f130dc33c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:57.344982 26340 tablet_bootstrap.cc:492] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f: Bootstrap starting.
I20260927 10:54:57.347774 26340 tablet_bootstrap.cc:654] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:57.348765 26333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.324904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:35085 (local address 127.0.0.1:35370)
0927 10:54:57.329671 (+ 4767us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:57.329684 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:57.329753 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:57.344531 (+ 14778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:57.344535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:57.344553 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:57.344642 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:57.344648 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.346000 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.346005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:57.347193 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:57.347204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.348366 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.348373 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:57.348467 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:57.348493 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 10:54:57.348550 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 10:54:57.348626 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4717,"thread_start_us":78,"threads_started":1}
I20260927 10:54:57.349196 26340 log.cc:824] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:57.349768 26338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.325299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.14.68.2:35085)
0927 10:54:57.344147 (+ 18848us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:57.344156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:57.344177 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:54:57.344238 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:57.344242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:57.344386 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:57.344557 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:57.344774 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.345850 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.347369 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.348234 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.349328 (+ 1094us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:57.349471 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:57.349523 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 10:54:57.349628 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18749,"thread_start_us":110,"threads_started":1}
I20260927 10:54:57.349942 26339 tablet_bootstrap.cc:492] T a7fe3ae481ca401e9061542f130dc33c P df5634aa9bb141518ab3877ee74eafec: Bootstrap starting.
I20260927 10:54:57.352154 26339 tablet_bootstrap.cc:654] T a7fe3ae481ca401e9061542f130dc33c P df5634aa9bb141518ab3877ee74eafec: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:57.352916 26339 log.cc:824] T a7fe3ae481ca401e9061542f130dc33c P df5634aa9bb141518ab3877ee74eafec: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:57.352802 26118 tablet_service.cc:1467] Processing CreateTablet for tablet a7fe3ae481ca401e9061542f130dc33c (DEFAULT_TABLE table=test-table [id=f1c757e43fa64b8cb4efb6f5742838d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:57.353863 26118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7fe3ae481ca401e9061542f130dc33c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:57.355012 26340 tablet_bootstrap.cc:492] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f: No bootstrap required, opened a new log
I20260927 10:54:57.355185 26340 ts_tablet_manager.cc:1397] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20260927 10:54:57.359068 26340 raft_consensus.cc:348] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec74237395f459692f6053170bad74f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45073 } } peers { permanent_uuid: "df5634aa9bb141518ab3877ee74eafec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 42793 } } peers { permanent_uuid: "5c459bdd257f4640981d97f0880921f7" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 35085 } }
I20260927 10:54:57.359326 26340 raft_consensus.cc:374] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:57.359376 26340 raft_consensus.cc:729] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec74237395f459692f6053170bad74f, State: Initialized, Role: FOLLOWER
I20260927 10:54:57.361164 26340 consensus_queue.cc:260] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec74237395f459692f6053170bad74f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45073 } } peers { permanent_uuid: "df5634aa9bb141518ab3877ee74eafec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 42793 } } peers { permanent_uuid: "5c459bdd257f4640981d97f0880921f7" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 35085 } }
I20260927 10:54:57.362224 26340 ts_tablet_manager.cc:1428] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 10:54:57.362241 26321 heartbeater.cc:503] Master 127.14.68.62:43929 was elected leader, sending a full tablet report...
I20260927 10:54:57.363678 26343 tablet_bootstrap.cc:492] T a7fe3ae481ca401e9061542f130dc33c P 5c459bdd257f4640981d97f0880921f7: Bootstrap starting.
I20260927 10:54:57.364140 26339 tablet_bootstrap.cc:492] T a7fe3ae481ca401e9061542f130dc33c P df5634aa9bb141518ab3877ee74eafec: No bootstrap required, opened a new log
I20260927 10:54:57.364307 26339 ts_tablet_manager.cc:1397] T a7fe3ae481ca401e9061542f130dc33c P df5634aa9bb141518ab3877ee74eafec: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20260927 10:54:57.366284 26343 tablet_bootstrap.cc:654] T a7fe3ae481ca401e9061542f130dc33c P 5c459bdd257f4640981d97f0880921f7: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:57.367020 26343 log.cc:824] T a7fe3ae481ca401e9061542f130dc33c P 5c459bdd257f4640981d97f0880921f7: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:57.368903 26343 tablet_bootstrap.cc:492] T a7fe3ae481ca401e9061542f130dc33c P 5c459bdd257f4640981d97f0880921f7: No bootstrap required, opened a new log
I20260927 10:54:57.369095 26343 ts_tablet_manager.cc:1397] T a7fe3ae481ca401e9061542f130dc33c P 5c459bdd257f4640981d97f0880921f7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:54:57.371637 26339 raft_consensus.cc:348] T a7fe3ae481ca401e9061542f130dc33c P df5634aa9bb141518ab3877ee74eafec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec74237395f459692f6053170bad74f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45073 } } peers { permanent_uuid: "df5634aa9bb141518ab3877ee74eafec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 42793 } } peers { permanent_uuid: "5c459bdd257f4640981d97f0880921f7" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 35085 } }
I20260927 10:54:57.371935 26339 raft_consensus.cc:374] T a7fe3ae481ca401e9061542f130dc33c P df5634aa9bb141518ab3877ee74eafec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:57.372006 26339 raft_consensus.cc:729] T a7fe3ae481ca401e9061542f130dc33c P df5634aa9bb141518ab3877ee74eafec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df5634aa9bb141518ab3877ee74eafec, State: Initialized, Role: FOLLOWER
I20260927 10:54:57.372419 26339 consensus_queue.cc:260] T a7fe3ae481ca401e9061542f130dc33c P df5634aa9bb141518ab3877ee74eafec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec74237395f459692f6053170bad74f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45073 } } peers { permanent_uuid: "df5634aa9bb141518ab3877ee74eafec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 42793 } } peers { permanent_uuid: "5c459bdd257f4640981d97f0880921f7" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 35085 } }
I20260927 10:54:57.372548 26344 raft_consensus.cc:482] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:54:57.372678 26344 raft_consensus.cc:504] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec74237395f459692f6053170bad74f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45073 } } peers { permanent_uuid: "df5634aa9bb141518ab3877ee74eafec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 42793 } } peers { permanent_uuid: "5c459bdd257f4640981d97f0880921f7" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 35085 } }
I20260927 10:54:57.373432 26339 ts_tablet_manager.cc:1428] T a7fe3ae481ca401e9061542f130dc33c P df5634aa9bb141518ab3877ee74eafec: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260927 10:54:57.373761 26344 leader_election.cc:288] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df5634aa9bb141518ab3877ee74eafec (127.14.68.1:42793), 5c459bdd257f4640981d97f0880921f7 (127.14.68.2:35085)
I20260927 10:54:57.373782 26049 heartbeater.cc:503] Master 127.14.68.62:43929 was elected leader, sending a full tablet report...
I20260927 10:54:57.376585 26343 raft_consensus.cc:348] T a7fe3ae481ca401e9061542f130dc33c P 5c459bdd257f4640981d97f0880921f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec74237395f459692f6053170bad74f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45073 } } peers { permanent_uuid: "df5634aa9bb141518ab3877ee74eafec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 42793 } } peers { permanent_uuid: "5c459bdd257f4640981d97f0880921f7" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 35085 } }
I20260927 10:54:57.376891 26343 raft_consensus.cc:374] T a7fe3ae481ca401e9061542f130dc33c P 5c459bdd257f4640981d97f0880921f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:57.376974 26343 raft_consensus.cc:729] T a7fe3ae481ca401e9061542f130dc33c P 5c459bdd257f4640981d97f0880921f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c459bdd257f4640981d97f0880921f7, State: Initialized, Role: FOLLOWER
I20260927 10:54:57.377419 26343 consensus_queue.cc:260] T a7fe3ae481ca401e9061542f130dc33c P 5c459bdd257f4640981d97f0880921f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec74237395f459692f6053170bad74f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45073 } } peers { permanent_uuid: "df5634aa9bb141518ab3877ee74eafec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 42793 } } peers { permanent_uuid: "5c459bdd257f4640981d97f0880921f7" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 35085 } }
I20260927 10:54:57.378414 26343 ts_tablet_manager.cc:1428] T a7fe3ae481ca401e9061542f130dc33c P 5c459bdd257f4640981d97f0880921f7: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260927 10:54:57.378800 26187 heartbeater.cc:503] Master 127.14.68.62:43929 was elected leader, sending a full tablet report...
I20260927 10:54:57.378851 26323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.374095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:35085 (local address 127.14.68.3:45899)
0927 10:54:57.374318 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:57.374332 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:57.374408 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:57.374980 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:57.374984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:57.375001 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:57.375134 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:57.375141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.376178 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.376182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:57.377353 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:57.377363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.378437 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.378441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:57.378546 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:57.378569 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:54:57.378643 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 10:54:57.378718 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260927 10:54:57.379462 26338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.374546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:45899 (local address 127.14.68.2:35085)
0927 10:54:57.374661 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:57.374666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:57.374680 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:54:57.374727 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:57.374733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:57.374791 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:57.374859 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:57.375255 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.376021 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.377646 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.378215 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.378316 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:57.378376 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:57.379267 (+ 891us) server_negotiation.cc:299] Negotiation successful
0927 10:54:57.379350 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:54:57.380051 26131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fe3ae481ca401e9061542f130dc33c" candidate_uuid: "5ec74237395f459692f6053170bad74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c459bdd257f4640981d97f0880921f7" is_pre_election: true
I20260927 10:54:57.380308 26131 raft_consensus.cc:2454] T a7fe3ae481ca401e9061542f130dc33c P 5c459bdd257f4640981d97f0880921f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ec74237395f459692f6053170bad74f in term 0.
I20260927 10:54:57.380823 26208 leader_election.cc:302] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c459bdd257f4640981d97f0880921f7, 5ec74237395f459692f6053170bad74f; no voters:
I20260927 10:54:57.382509 26344 raft_consensus.cc:2781] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:54:57.382608 26344 raft_consensus.cc:482] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:54:57.382655 26344 raft_consensus.cc:3037] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:57.385185 26323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.374998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:42793 (local address 127.14.68.3:38729)
0927 10:54:57.378995 (+ 3997us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:57.379008 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:57.379101 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:57.379429 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:57.379432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:57.379448 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:57.379530 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:57.379535 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.382485 (+ 2950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.382489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:57.383583 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:57.383593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.384783 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.384787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:57.384873 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:57.384898 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:54:57.384973 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 10:54:57.385037 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3948,"thread_start_us":96,"threads_started":1}
I20260927 10:54:57.385591 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.375499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:38729 (local address 127.14.68.1:42793)
0927 10:54:57.375634 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:57.375639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:57.379105 (+ 3466us) server_negotiation.cc:407] Connection header received
0927 10:54:57.379220 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:57.379223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:57.379272 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:57.379335 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:57.381607 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.382356 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.384020 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.384655 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.384768 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:57.384826 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:57.385400 (+ 574us) server_negotiation.cc:299] Negotiation successful
0927 10:54:57.385474 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:54:57.386458 25990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fe3ae481ca401e9061542f130dc33c" candidate_uuid: "5ec74237395f459692f6053170bad74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5634aa9bb141518ab3877ee74eafec" is_pre_election: true
I20260927 10:54:57.386741 25990 raft_consensus.cc:2454] T a7fe3ae481ca401e9061542f130dc33c P df5634aa9bb141518ab3877ee74eafec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ec74237395f459692f6053170bad74f in term 0.
I20260927 10:54:57.385291 26344 raft_consensus.cc:504] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec74237395f459692f6053170bad74f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45073 } } peers { permanent_uuid: "df5634aa9bb141518ab3877ee74eafec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 42793 } } peers { permanent_uuid: "5c459bdd257f4640981d97f0880921f7" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 35085 } }
I20260927 10:54:57.387627 26344 leader_election.cc:288] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f [CANDIDATE]: Term 1 election: Requested vote from peers df5634aa9bb141518ab3877ee74eafec (127.14.68.1:42793), 5c459bdd257f4640981d97f0880921f7 (127.14.68.2:35085)
I20260927 10:54:57.388125 25990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fe3ae481ca401e9061542f130dc33c" candidate_uuid: "5ec74237395f459692f6053170bad74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5634aa9bb141518ab3877ee74eafec"
I20260927 10:54:57.388181 26131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fe3ae481ca401e9061542f130dc33c" candidate_uuid: "5ec74237395f459692f6053170bad74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c459bdd257f4640981d97f0880921f7"
I20260927 10:54:57.388236 25990 raft_consensus.cc:3037] T a7fe3ae481ca401e9061542f130dc33c P df5634aa9bb141518ab3877ee74eafec [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:57.388300 26131 raft_consensus.cc:3037] T a7fe3ae481ca401e9061542f130dc33c P 5c459bdd257f4640981d97f0880921f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:57.389497 25990 raft_consensus.cc:2454] T a7fe3ae481ca401e9061542f130dc33c P df5634aa9bb141518ab3877ee74eafec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ec74237395f459692f6053170bad74f in term 1.
I20260927 10:54:57.389565 26131 raft_consensus.cc:2454] T a7fe3ae481ca401e9061542f130dc33c P 5c459bdd257f4640981d97f0880921f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ec74237395f459692f6053170bad74f in term 1.
I20260927 10:54:57.389865 26210 leader_election.cc:302] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec74237395f459692f6053170bad74f, df5634aa9bb141518ab3877ee74eafec; no voters:
I20260927 10:54:57.390097 26344 raft_consensus.cc:2781] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:57.390331 26344 raft_consensus.cc:686] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f [term 1 LEADER]: Becoming Leader. State: Replica: 5ec74237395f459692f6053170bad74f, State: Running, Role: LEADER
I20260927 10:54:57.390688 26344 consensus_queue.cc:237] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec74237395f459692f6053170bad74f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45073 } } peers { permanent_uuid: "df5634aa9bb141518ab3877ee74eafec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 42793 } } peers { permanent_uuid: "5c459bdd257f4640981d97f0880921f7" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 35085 } }
I20260927 10:54:57.395751 25837 catalog_manager.cc:5495] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ec74237395f459692f6053170bad74f (127.14.68.3). New cstate: current_term: 1 leader_uuid: "5ec74237395f459692f6053170bad74f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec74237395f459692f6053170bad74f" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df5634aa9bb141518ab3877ee74eafec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 42793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c459bdd257f4640981d97f0880921f7" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 35085 } health_report { overall_health: UNKNOWN } } }
I20260927 10:54:57.409143 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.404421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36160 (local address 127.14.68.3:45073)
0927 10:54:57.404529 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:57.404533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:57.404623 (+ 90us) server_negotiation.cc:407] Connection header received
0927 10:54:57.404775 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:57.404778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:57.404844 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:57.404952 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:57.405276 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.406094 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.407317 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:57.407663 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:57.407797 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:57.408121 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:57.408672 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:57.408817 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:57.408931 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 10:54:57.408992 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 10:54:57.416062 26131 raft_consensus.cc:1261] T a7fe3ae481ca401e9061542f130dc33c P 5c459bdd257f4640981d97f0880921f7 [term 1 FOLLOWER]: Refusing update from remote peer 5ec74237395f459692f6053170bad74f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:57.416103 25990 raft_consensus.cc:1261] T a7fe3ae481ca401e9061542f130dc33c P df5634aa9bb141518ab3877ee74eafec [term 1 FOLLOWER]: Refusing update from remote peer 5ec74237395f459692f6053170bad74f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:57.416851 26349 consensus_queue.cc:1035] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f [LEADER]: Connected to new peer: Peer: permanent_uuid: "df5634aa9bb141518ab3877ee74eafec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 42793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:57.417008 26344 consensus_queue.cc:1035] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c459bdd257f4640981d97f0880921f7" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 35085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:57.430172 26352 mvcc.cc:205] Tried to move back new op lower bound from 7333914613407940608 to 7333914613315260416. Current Snapshot: MvccSnapshot[applied={T|T < 7333914613407940608}]
I20260927 10:54:57.431365 26354 mvcc.cc:205] Tried to move back new op lower bound from 7333914613407940608 to 7333914613315260416. Current Snapshot: MvccSnapshot[applied={T|T < 7333914613407940608}]
W20260927 10:54:57.432258 26188 tablet.cc:2369] T a7fe3ae481ca401e9061542f130dc33c P 5c459bdd257f4640981d97f0880921f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:54:57.443552 26353 mvcc.cc:205] Tried to move back new op lower bound from 7333914613407940608 to 7333914613315260416. Current Snapshot: MvccSnapshot[applied={T|T < 7333914613407940608}]
I20260927 10:54:57.445061 25836 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:54:57.445233 25836 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:54:57.447986 25836 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0971230edc248058a4eeeee7451620c: Sending DeleteTablet for 3 replicas of tablet a7fe3ae481ca401e9061542f130dc33c
I20260927 10:54:57.448894 26118 tablet_service.cc:1514] Processing DeleteTablet for tablet a7fe3ae481ca401e9061542f130dc33c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:57 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:35370
I20260927 10:54:57.449689 26255 tablet_service.cc:1514] Processing DeleteTablet for tablet a7fe3ae481ca401e9061542f130dc33c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:57 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:36156
I20260927 10:54:57.450281 25983 tablet_service.cc:1514] Processing DeleteTablet for tablet a7fe3ae481ca401e9061542f130dc33c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:57 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:43396
I20260927 10:54:57.450999 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 25918
I20260927 10:54:57.452296 26361 tablet_replica.cc:330] T a7fe3ae481ca401e9061542f130dc33c P 5c459bdd257f4640981d97f0880921f7: stopping tablet replica
I20260927 10:54:57.452590 26361 raft_consensus.cc:2229] T a7fe3ae481ca401e9061542f130dc33c P 5c459bdd257f4640981d97f0880921f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:54:57.452879 26361 raft_consensus.cc:2258] T a7fe3ae481ca401e9061542f130dc33c P 5c459bdd257f4640981d97f0880921f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:57.453228 26363 tablet_replica.cc:330] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f: stopping tablet replica
I20260927 10:54:57.453474 26363 raft_consensus.cc:2229] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:54:57.453840 26363 raft_consensus.cc:2258] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:57.455284 26363 ts_tablet_manager.cc:1905] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:54:57.458770 26363 ts_tablet_manager.cc:1918] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:54:57.458858 26363 log.cc:1196] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/a7fe3ae481ca401e9061542f130dc33c
I20260927 10:54:57.459180 26363 ts_tablet_manager.cc:1939] T a7fe3ae481ca401e9061542f130dc33c P 5ec74237395f459692f6053170bad74f: Deleting consensus metadata
I20260927 10:54:57.460861 26361 ts_tablet_manager.cc:1905] T a7fe3ae481ca401e9061542f130dc33c P 5c459bdd257f4640981d97f0880921f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:54:57.460871 25822 catalog_manager.cc:4841] TS 5ec74237395f459692f6053170bad74f (127.14.68.3:45073): tablet a7fe3ae481ca401e9061542f130dc33c (table test-table [id=f1c757e43fa64b8cb4efb6f5742838d2]) successfully deleted
W20260927 10:54:57.461585 25823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.68.1:42793 (error 108)
I20260927 10:54:57.462370 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 26057
I20260927 10:54:57.466084 26333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.465723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:42793 (local address 127.0.0.1:43404)
0927 10:54:57.465871 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:57.465953 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.1:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 10:54:57.466243 25823 catalog_manager.cc:4584] TS df5634aa9bb141518ab3877ee74eafec (127.14.68.1:42793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a7fe3ae481ca401e9061542f130dc33c: Network error: Client connection negotiation failed: client connection to 127.14.68.1:42793: connect: Connection refused (error 111)
W20260927 10:54:57.469952 26333 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:57.469357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:35085 (local address 127.0.0.1:35372)
0927 10:54:57.469514 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:57.469528 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:57.469628 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:57.469852 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.2:35085: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 10:54:57.470587 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 26192
I20260927 10:54:57.478322 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 25805
I20260927 10:54:57.485535 14608 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 10:54:57.830138 14608 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 10:54:57.830259 14608 mini_ranger.cc:66] Stopping Ranger...
I20260927 10:54:58.175127 14608 mini_ranger.cc:68] Stopped Ranger
2026-09-27 10:54:58.175 UTC [25391] LOG: received smart shutdown request
2026-09-27 10:54:58.178 UTC [25391] LOG: background worker "logical replication launcher" (PID 25400) exited with exit code 1
2026-09-27 10:54:58.179 UTC [25395] LOG: shutting down
2026-09-27 10:54:58.365 UTC [25391] LOG: database system is shut down
2026-09-27T10:54:58Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46538 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:58 dist-test-slave-tz51 krb5kdc[26374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:58 dist-test-slave-tz51 krb5kdc[26374](info): set up 2 sockets
Sep 27 10:54:58 dist-test-slave-tz51 krb5kdc[26374](info): commencing operation
krb5kdc: starting...
W20260927 10:54:59.507759 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:54:59 dist-test-slave-tz51 krb5kdc[26374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.68.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.68.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.68.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/rangeradmin_127.14.68.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.68.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/rangeradmin_127.14.68.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.68.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.68.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.68.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/rangerlookup_127.14.68.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.68.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/rangerlookup_127.14.68.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.68.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.68.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.68.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/HTTP_127.14.68.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.68.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/HTTP_127.14.68.20@KRBTEST.COM.keytab.
I20260927 10:54:59.676234 14608 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskAZZUsq/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/postgres -l logfile start
2026-09-27 10:55:02.179 UTC [26404] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 10:55:02.180 UTC [26404] LOG: listening on IPv4 address "127.14.68.20", port 46327
2026-09-27 10:55:02.183 UTC [26404] LOG: listening on Unix socket "/tmp/.s.PGSQL.46327"
2026-09-27 10:55:02.217 UTC [26407] LOG: database system was shut down at 2026-09-27 10:55:00 UTC
2026-09-27 10:55:02.222 UTC [26404] LOG: database system is ready to accept connections
I20260927 10:55:03.163656 14608 mini_postgres.cc:96] Postgres bound to 46327
2026-09-27 10:55:03.169 UTC [26415] FATAL: database "slave" does not exist
127.14.68.20:46327 - accepting connections
I20260927 10:55:03.170678 14608 mini_ranger.cc:161] Starting Ranger...
I20260927 10:55:03.185719 14608 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 10:55:03.680399 14608 mini_ranger.cc:87] Created ranger Postgres database
I20260927 10:55:03.680488 14608 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ranger-admin
2026-09-27 10:55:03,960 [I] DB FLAVOR :POSTGRES
2026-09-27 10:55:03,961 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 10:55:03,961 [I] Checking connection..
2026-09-27 10:55:03,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:46327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 10:55:04,223 [I] Checking connection passed.
2026-09-27 10:55:04,223 [I] --------- Verifying version history table ---------
2026-09-27 10:55:04,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:46327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:55:04,455 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 10:55:04,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:46327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:55:04,677 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 10:55:04,677 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 10:55:04,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:46327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 10:55:04.910 UTC [26479] WARNING: there is no transaction in progress
2026-09-27 10:55:04,916 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 10:55:04,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:46327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:55:05,163 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 10:55:05,164 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 10:55:05,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:46327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 10:55:05,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:46327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 10:55:05,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:46327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tz51.c.gcp-upstream.internal','N') ;"
2026-09-27 10:55:05,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:46327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 10:55:06,106 [I] Table x_portal_user does not exist in database ranger
2026-09-27 10:55:06,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:46327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 10:55:06,333 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 10:55:06,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:46327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 10:55:06,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:46327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 10:55:06,792 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 10:55:06,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:46327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 10:55:07.754 UTC [26587] WARNING: there is no transaction in progress
2026-09-27 10:55:07.771 UTC [26587] WARNING: there is no transaction in progress
2026-09-27 10:55:07.786 UTC [26587] WARNING: there is no transaction in progress
2026-09-27 10:55:07.802 UTC [26587] WARNING: there is no transaction in progress
2026-09-27 10:55:07.818 UTC [26587] WARNING: there is no transaction in progress
2026-09-27 10:55:07.836 UTC [26587] WARNING: there is no transaction in progress
2026-09-27 10:55:07.898 UTC [26587] WARNING: there is no transaction in progress
2026-09-27 10:55:07.904 UTC [26587] WARNING: there is no transaction in progress
2026-09-27 10:55:07.918 UTC [26587] WARNING: there is no transaction in progress
2026-09-27 10:55:07.928 UTC [26587] WARNING: there is no transaction in progress
2026-09-27 10:55:07.939 UTC [26587] WARNING: there is no transaction in progress
2026-09-27 10:55:07.954 UTC [26587] WARNING: there is no transaction in progress
2026-09-27 10:55:07.963 UTC [26587] WARNING: there is no transaction in progress
2026-09-27 10:55:07.969 UTC [26587] WARNING: there is no transaction in progress
2026-09-27 10:55:07.976 UTC [26587] WARNING: there is no transaction in progress
2026-09-27 10:55:07.983 UTC [26587] WARNING: there is no transaction in progress
2026-09-27 10:55:08,612 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 10:55:08,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:46327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 10:55:08,833 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 10:55:08,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:46327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 10:55:09,075 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 10:55:09,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:46327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 10:55:09,333 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 10:55:09,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:46327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 10:55:09,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:46327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 10:55:09,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:46327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tz51.c.gcp-upstream.internal';"
2026-09-27 10:55:10,054 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 10:55:10,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:46327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 10:55:10,304 [I] DB_PATCHES have already been applied
I20260927 10:55:10.309857 14608 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ranger-admin:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskAZZUsq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 10:55:10.309934 14608 mini_ranger.cc:193] Using host: 127.14.68.20
I20260927 10:55:10.313442 14608 mini_ranger.cc:227] Ranger admin URL: http://127.14.68.20:39229
Sep 27, 2026 10:55:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:55:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:55:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:55:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:55:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:55:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:55:11 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 10:55:11.496760 14608 mini_ranger.cc:160] Time spent starting Ranger: real 8.326s user 0.000s sys 0.009s
Sep 27, 2026 10:55:32 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:55:32 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 10:55:32 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:55:32 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:55:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:55:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:55:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:55:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:55:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:55:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:55:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:55:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:55:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:55:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:55:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:55:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:55:32 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:55:33 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 10:55:33.697839 14608 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.68.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.68.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.68.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/rangerkms_127.14.68.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.68.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/rangerkms_127.14.68.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.68.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.68.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.68.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/HTTP_127.14.68.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.68.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/HTTP_127.14.68.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 10:55:33 dist-test-slave-tz51 krb5kdc[26374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506533, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 10:55:33.786450 14608 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 10:55:33.795650 14608 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 10:55:34.718995 14608 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 10:55:34.719130 14608 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ranger-kms
I20260927 10:55:34.719158 14608 mini_ranger_kms.cc:227] Using postgres at 127.14.68.20:46327
2026-09-27 10:55:34,891 [I] DB FLAVOR :POSTGRES
2026-09-27 10:55:34,891 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 10:55:34,892 [I] Checking connection
2026-09-27 10:55:34,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:46327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 10:55:35,135 [I] connection success
2026-09-27 10:55:35,135 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 10:55:35,136 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 10:55:35,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:46327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 10:55:35,370 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 10:55:35,371 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 10:55:35,371 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 10:55:35,371 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.68.20:46327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 10:55:35,657 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 10:55:36.135330 14608 mini_ranger_kms.cc:310] Created kms service
I20260927 10:55:36.332608 14608 mini_ranger_kms.cc:326] Created kudu user
I20260927 10:55:36.375519 14608 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 10:55:36 dist-test-slave-tz51 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506533, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.68.20@KRBTEST.COM
I20260927 10:55:36.776784 14608 mini_ranger_kms.cc:384] Added ranger policy
I20260927 10:55:36.776930 14608 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ranger-kms:/tmp/dist-test-taskAZZUsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskAZZUsq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 10:55:36.776952 14608 mini_ranger_kms.cc:240] Using host: 127.14.68.20
I20260927 10:55:36.780733 14608 mini_ranger_kms.cc:276] Ranger KMS PID: 26767
I20260927 10:55:36.780817 14608 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.68.20:47007
Sep 27, 2026 10:55:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:55:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:55:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:55:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:55:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:55:38 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 10:55:39.056136 14608 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.270s user 0.013s sys 0.144s
I20260927 10:55:39.056267 14608 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 10:55:39.056339 14608 mini_ranger_kms.cc:401] 127.14.68.20:47007/kms/v1/keys
Sep 27 10:55:39 dist-test-slave-tz51 krb5kdc[26374](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790506539, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.68.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 10:55:39 dist-test-slave-tz51 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506539, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.68.20@KRBTEST.COM for HTTP/127.14.68.20@KRBTEST.COM
Sep 27 10:55:39 dist-test-slave-tz51 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506539, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.68.20@KRBTEST.COM for HTTP/127.14.68.20@KRBTEST.COM
Sep 27 10:55:41 dist-test-slave-tz51 krb5kdc[26374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:55:42.035177 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:46883
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:46883
--ranger_config_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.68.20:47007/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:42.157493 26818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:42.157994 26818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:42.158073 26818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:42.162084 26818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:55:42.162139 26818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:42.162159 26818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:42.162175 26818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:55:42.162190 26818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:55:42.167793 26818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.68.20:47007/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:46883
--ranger_config_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:46883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:42.168749 26818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:42.169685 26818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:42.178058 26823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:42.178049 26824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:42.178094 26818 server_base.cc:1034] running on GCE node
W20260927 10:55:42.178032 26826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:42.178890 26818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:42.180002 26818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:42.181165 26818 hybrid_clock.cc:648] HybridClock initialized: now 1790506542181147 us; error 32 us; skew 500 ppm
Sep 27 10:55:42 dist-test-slave-tz51 krb5kdc[26374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506542, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:42.184042 26818 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:55:42.185020 26818 webserver.cc:460] Webserver started at http://127.14.68.62:44023/ using document root <none> and password file <none>
I20260927 10:55:42.185568 26818 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:42.185622 26818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:42.185824 26818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:55:42 dist-test-slave-tz51 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506542, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for HTTP/127.14.68.20@KRBTEST.COM
I20260927 10:55:42.232187 26818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "4c261ce3483940efbd6fc2503a460757"
format_stamp: "Formatted at 2026-09-27 10:55:42 on dist-test-slave-tz51"
server_key: "a46a4a69a3329d41744217fcec1c9512"
server_key_iv: "6736b046e5e4a2b3ba917cdb39ae023c"
server_key_version: "kuduclusterkey@0"
I20260927 10:55:42.232839 26818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "4c261ce3483940efbd6fc2503a460757"
format_stamp: "Formatted at 2026-09-27 10:55:42 on dist-test-slave-tz51"
server_key: "a46a4a69a3329d41744217fcec1c9512"
server_key_iv: "6736b046e5e4a2b3ba917cdb39ae023c"
server_key_version: "kuduclusterkey@0"
I20260927 10:55:42.236754 26818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:55:42.251984 26833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:42.253638 26818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:55:42.253768 26818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "4c261ce3483940efbd6fc2503a460757"
format_stamp: "Formatted at 2026-09-27 10:55:42 on dist-test-slave-tz51"
server_key: "a46a4a69a3329d41744217fcec1c9512"
server_key_iv: "6736b046e5e4a2b3ba917cdb39ae023c"
server_key_version: "kuduclusterkey@0"
I20260927 10:55:42.253896 26818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:42.321745 26818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:42.325304 26818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:42.325508 26818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:42.334188 26885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:46883 every 8 connection(s)
I20260927 10:55:42.334197 26818 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:46883
I20260927 10:55:42.335407 26818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:55:42.338320 26886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:42.343966 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 26818
I20260927 10:55:42.344105 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:55:42.344204 26886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757: Bootstrap starting.
I20260927 10:55:42.344455 14608 external_mini_cluster.cc:1428] Setting key 8e4060438918b76b5e683dd6c636bf38
I20260927 10:55:42.347024 26886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:42.347936 26886 log.cc:824] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:42.349962 26886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757: No bootstrap required, opened a new log
Sep 27 10:55:42 dist-test-slave-tz51 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:55:42.354404 26886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c261ce3483940efbd6fc2503a460757" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46883 } }
I20260927 10:55:42.354655 26886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:42.354774 26886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c261ce3483940efbd6fc2503a460757, State: Initialized, Role: FOLLOWER
I20260927 10:55:42.355305 26886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c261ce3483940efbd6fc2503a460757" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46883 } }
I20260927 10:55:42.355474 26886 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:55:42.355553 26886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:55:42.355664 26886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:42.356546 26886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c261ce3483940efbd6fc2503a460757" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46883 } }
I20260927 10:55:42.356916 26886 leader_election.cc:302] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c261ce3483940efbd6fc2503a460757; no voters:
I20260927 10:55:42.357299 26886 leader_election.cc:288] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:55:42.357403 26891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:55:42.357630 26891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [term 1 LEADER]: Becoming Leader. State: Replica: 4c261ce3483940efbd6fc2503a460757, State: Running, Role: LEADER
I20260927 10:55:42.357992 26891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c261ce3483940efbd6fc2503a460757" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46883 } }
I20260927 10:55:42.358646 26886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:55:42.359735 26892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c261ce3483940efbd6fc2503a460757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c261ce3483940efbd6fc2503a460757" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46883 } } }
I20260927 10:55:42.359861 26892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:42.360194 26893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c261ce3483940efbd6fc2503a460757. Latest consensus state: current_term: 1 leader_uuid: "4c261ce3483940efbd6fc2503a460757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c261ce3483940efbd6fc2503a460757" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46883 } } }
I20260927 10:55:42.360285 26893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:42.360523 26889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:42.345928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.14.68.62:46883)
0927 10:55:42.346397 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:42.346413 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:42.346448 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:55:42.347144 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:42.347175 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:42.347539 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:42.347896 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:42.348762 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:42.349756 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:42.350938 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:42.351423 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:42.351552 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:42.354405 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:42.354410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:42.354423 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:42.354448 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:42.356316 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:42.356763 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:42.356769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:42.356775 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:42.356900 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:42.357211 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:42.357218 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:42.357222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:42.357746 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:42.358613 (+ 867us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:42.358811 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 10:55:42.359207 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":178,"threads_started":1}
I20260927 10:55:42.360823 26886 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 10:55:42.360550 26900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:55:42.366971 26900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:55:42.373539 26900 catalog_manager.cc:1347] Generated new cluster ID: ea7dfda805da441fa4b69b4edc24dd01
I20260927 10:55:42.373611 26900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:55:42.379410 26900 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:55:42.380406 26900 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:55:42.393175 26900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757: Generated new TSK 0
I20260927 10:55:42.394266 26900 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
I20260927 10:55:43.051236 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.68.20:47007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46883
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:43.168210 26931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:43.168474 26931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:43.168548 26931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:43.173058 26931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:43.173133 26931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:43.173216 26931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:55:43.178959 26931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.68.20:47007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:43.180262 26931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:43.181267 26931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:43.191226 26937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:43.191435 26939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:43.191216 26936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:43.191298 26931 server_base.cc:1034] running on GCE node
I20260927 10:55:43.192996 26931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:43.193593 26931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:43.194798 26931 hybrid_clock.cc:648] HybridClock initialized: now 1790506543194771 us; error 44 us; skew 500 ppm
Sep 27 10:55:43 dist-test-slave-tz51 krb5kdc[26374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506543, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:43.198493 26931 init.cc:377] Logged in from keytab as kudu/127.14.68.1@KRBTEST.COM (short username kudu)
I20260927 10:55:43.199663 26931 webserver.cc:460] Webserver started at http://127.14.68.1:44307/ using document root <none> and password file <none>
I20260927 10:55:43.200450 26931 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:43.200538 26931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:43.200845 26931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:55:43 dist-test-slave-tz51 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506543, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for HTTP/127.14.68.20@KRBTEST.COM
I20260927 10:55:43.218508 26931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "eedbc635919f42018cfacbf15ec2c43d"
format_stamp: "Formatted at 2026-09-27 10:55:43 on dist-test-slave-tz51"
server_key: "fb124f475946f32c4689d174a54854d4"
server_key_iv: "27e6b5f03f738c63cec4bb31e14d2611"
server_key_version: "kuduclusterkey@0"
I20260927 10:55:43.219341 26931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "eedbc635919f42018cfacbf15ec2c43d"
format_stamp: "Formatted at 2026-09-27 10:55:43 on dist-test-slave-tz51"
server_key: "fb124f475946f32c4689d174a54854d4"
server_key_iv: "27e6b5f03f738c63cec4bb31e14d2611"
server_key_version: "kuduclusterkey@0"
I20260927 10:55:43.225455 26931 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.000s
I20260927 10:55:43.244063 26948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:43.245705 26931 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260927 10:55:43.245887 26931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "eedbc635919f42018cfacbf15ec2c43d"
format_stamp: "Formatted at 2026-09-27 10:55:43 on dist-test-slave-tz51"
server_key: "fb124f475946f32c4689d174a54854d4"
server_key_iv: "27e6b5f03f738c63cec4bb31e14d2611"
server_key_version: "kuduclusterkey@0"
I20260927 10:55:43.246106 26931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:43.266723 26931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:43.271386 26931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:43.271647 26931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:43.272429 26931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:43.273488 26931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:43.273598 26931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:43.273744 26931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:43.273842 26931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:43.294071 26931 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:34977
I20260927 10:55:43.295332 26931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:55:43.296821 27061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:34977 every 8 connection(s)
I20260927 10:55:43.298916 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 26931
I20260927 10:55:43.299027 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:55:43.299350 14608 external_mini_cluster.cc:1428] Setting key d138656d736cd9066ca3fb5e8f627efe
Sep 27 10:55:43 dist-test-slave-tz51 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506543, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.68.2@KRBTEST.COM; defaulting to no policy
I20260927 10:55:43.319828 27064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:43.299143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:46883 (local address 127.14.68.1:42653)
0927 10:55:43.301045 (+ 1902us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:43.301101 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:43.302325 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:43.303328 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:43.303337 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:43.303681 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:43.304373 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:43.304391 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:43.305532 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:43.305535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:43.306971 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:43.306981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:43.307806 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:43.307810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:43.308008 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:43.308700 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:43.308738 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:43.310535 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:43.312691 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:43.312698 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:43.312716 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:43.313090 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:43.313446 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:43.313450 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:43.313455 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:43.313653 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:43.314128 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:43.314137 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:43.318055 (+ 3918us) client_negotiation.cc:769] Sending connection context
0927 10:55:43.318326 (+ 271us) client_negotiation.cc:240] Negotiation successful
0927 10:55:43.318673 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1677,"thread_start_us":131,"threads_started":1}
I20260927 10:55:43.320580 27065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:43.300235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:42653 (local address 127.14.68.62:46883)
0927 10:55:43.302789 (+ 2554us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:43.302798 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:43.302811 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:55:43.302867 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:43.302870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:43.302915 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:43.302998 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:43.304539 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:43.305384 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:43.307114 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:43.307473 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:43.307561 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:43.310763 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:43.310768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:43.310771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:43.310800 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:43.312557 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:43.313196 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:43.313199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:43.313200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:43.313329 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:43.313765 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:43.313769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:43.313771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:43.313998 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:43.314113 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:43.320279 (+ 6166us) server_negotiation.cc:299] Negotiation successful
0927 10:55:43.320413 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2440,"thread_start_us":102,"threads_started":1}
I20260927 10:55:43.322651 27062 heartbeater.cc:347] Connected to a master server at 127.14.68.62:46883
I20260927 10:55:43.322891 27062 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:43.323460 27062 heartbeater.cc:511] Master 127.14.68.62:46883 requested a full tablet report, sending...
I20260927 10:55:43.325264 26850 ts_manager.cc:194] Registered new tserver with Master: eedbc635919f42018cfacbf15ec2c43d (127.14.68.1:34977)
I20260927 10:55:43.326853 26850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:42653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
I20260927 10:55:43.347792 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.68.20:47007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46883
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:55:43 dist-test-slave-tz51 krb5kdc[26374](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790506543, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 10:55:43.507194 27070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:43.507576 27070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:43.507647 27070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:43.513770 27070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:43.513855 27070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:43.513944 27070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:55:43.521829 27070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.68.20:47007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:43.523181 27070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:43.524591 27070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:43.536767 27079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:43.536790 27076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:43.537189 27077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:43.537803 27070 server_base.cc:1034] running on GCE node
I20260927 10:55:43.538250 27070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:43.539003 27070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:43.540195 27070 hybrid_clock.cc:648] HybridClock initialized: now 1790506543540154 us; error 56 us; skew 500 ppm
Sep 27 10:55:43 dist-test-slave-tz51 krb5kdc[26374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506543, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:43.543764 27070 init.cc:377] Logged in from keytab as kudu/127.14.68.2@KRBTEST.COM (short username kudu)
I20260927 10:55:43.544864 27070 webserver.cc:460] Webserver started at http://127.14.68.2:33967/ using document root <none> and password file <none>
I20260927 10:55:43.545586 27070 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:43.545687 27070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:43.545918 27070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:55:43 dist-test-slave-tz51 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506543, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for HTTP/127.14.68.20@KRBTEST.COM
I20260927 10:55:43.564735 27070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "8ad364827623419ea8e6185d45bd7a09"
format_stamp: "Formatted at 2026-09-27 10:55:43 on dist-test-slave-tz51"
server_key: "6bb89a57c7dbe3c3ff289b2a942ffa52"
server_key_iv: "b1532248284ef229742bd473eb3d97e3"
server_key_version: "kuduclusterkey@0"
I20260927 10:55:43.565634 27070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "8ad364827623419ea8e6185d45bd7a09"
format_stamp: "Formatted at 2026-09-27 10:55:43 on dist-test-slave-tz51"
server_key: "6bb89a57c7dbe3c3ff289b2a942ffa52"
server_key_iv: "b1532248284ef229742bd473eb3d97e3"
server_key_version: "kuduclusterkey@0"
I20260927 10:55:43.584620 27070 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.004s sys 0.000s
I20260927 10:55:43.612087 27086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:43.613986 27070 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260927 10:55:43.614218 27070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "8ad364827623419ea8e6185d45bd7a09"
format_stamp: "Formatted at 2026-09-27 10:55:43 on dist-test-slave-tz51"
server_key: "6bb89a57c7dbe3c3ff289b2a942ffa52"
server_key_iv: "b1532248284ef229742bd473eb3d97e3"
server_key_version: "kuduclusterkey@0"
I20260927 10:55:43.614532 27070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:43.659518 27070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:43.667735 27070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:43.667941 27070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:43.668707 27070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:43.669744 27070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:43.669806 27070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:43.669858 27070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:43.669889 27070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:43.680150 27070 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:38235
I20260927 10:55:43.681311 27070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:55:43.686471 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 27070
I20260927 10:55:43.686591 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:55:43.686903 14608 external_mini_cluster.cc:1428] Setting key 4192b07dedf1c9e9d502b100be05d078
I20260927 10:55:43.688076 27199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:38235 every 8 connection(s)
WARNING: no policy specified for kudu/127.14.68.3@KRBTEST.COM; defaulting to no policy
Sep 27 10:55:43 dist-test-slave-tz51 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506543, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.3@KRBTEST.COM" created.
I20260927 10:55:43.707267 27065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:43.693740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:37923 (local address 127.14.68.62:46883)
0927 10:55:43.693849 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:43.693853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:43.693869 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:55:43.694770 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:43.694776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:43.694834 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:43.694888 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:43.696594 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:43.697502 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:43.699403 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:43.699837 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:43.699923 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:43.703090 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:43.703094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:43.703097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:43.703130 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:43.704742 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:43.705277 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:43.705280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:43.705282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:43.705366 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:43.705772 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:43.705778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:43.705783 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:43.706021 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:43.706114 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:43.706854 (+ 740us) server_negotiation.cc:299] Negotiation successful
0927 10:55:43.707007 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 10:55:43.709060 27203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:43.692682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:46883 (local address 127.14.68.2:37923)
0927 10:55:43.693203 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:43.693268 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:43.694514 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:43.695336 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:43.695347 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:43.695690 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:43.696402 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:43.696425 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:43.697676 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:43.697682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:43.699264 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:43.699276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:43.700133 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:43.700137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:43.700375 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:43.701085 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:43.701113 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:43.702911 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:43.704869 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:43.704875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:43.704890 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:43.705145 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:43.705468 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:43.705472 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:43.705474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:43.705654 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:43.706206 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:43.706214 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:43.706566 (+ 352us) client_negotiation.cc:769] Sending connection context
0927 10:55:43.707503 (+ 937us) client_negotiation.cc:240] Negotiation successful
0927 10:55:43.707910 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":171,"threads_started":1}
I20260927 10:55:43.710358 27200 heartbeater.cc:347] Connected to a master server at 127.14.68.62:46883
I20260927 10:55:43.710603 27200 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:43.711131 27200 heartbeater.cc:511] Master 127.14.68.62:46883 requested a full tablet report, sending...
I20260927 10:55:43.712247 26850 ts_manager.cc:194] Registered new tserver with Master: 8ad364827623419ea8e6185d45bd7a09 (127.14.68.2:38235)
I20260927 10:55:43.713089 26850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:37923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
I20260927 10:55:43.726441 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.68.20:47007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46883
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:55:43 dist-test-slave-tz51 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506543, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for HTTP/127.14.68.20@KRBTEST.COM
W20260927 10:55:43.869145 27205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:43.869410 27205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:43.869453 27205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:43.875336 27205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:43.875424 27205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:43.875515 27205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:55:43.883613 27205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.68.20:47007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:43.885131 27205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:43.886529 27205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:43.898907 27210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:43.899441 27213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:43.899475 27205 server_base.cc:1034] running on GCE node
W20260927 10:55:43.899793 27211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:43.900292 27205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:43.901173 27205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:43.902383 27205 hybrid_clock.cc:648] HybridClock initialized: now 1790506543902355 us; error 44 us; skew 500 ppm
Sep 27 10:55:43 dist-test-slave-tz51 krb5kdc[26374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506543, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:43.906101 27205 init.cc:377] Logged in from keytab as kudu/127.14.68.3@KRBTEST.COM (short username kudu)
I20260927 10:55:43.907179 27205 webserver.cc:460] Webserver started at http://127.14.68.3:37617/ using document root <none> and password file <none>
I20260927 10:55:43.907909 27205 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:43.908023 27205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:43.908263 27205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:55:43 dist-test-slave-tz51 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506543, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for HTTP/127.14.68.20@KRBTEST.COM
I20260927 10:55:43.928617 27205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "3fcc3c258494451e838d3275917dedad"
format_stamp: "Formatted at 2026-09-27 10:55:43 on dist-test-slave-tz51"
server_key: "5616c772d546f41787b51fc92ebe3f2c"
server_key_iv: "bed4a896a43009e04deef7b2dac61330"
server_key_version: "kuduclusterkey@0"
I20260927 10:55:43.929392 27205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "3fcc3c258494451e838d3275917dedad"
format_stamp: "Formatted at 2026-09-27 10:55:43 on dist-test-slave-tz51"
server_key: "5616c772d546f41787b51fc92ebe3f2c"
server_key_iv: "bed4a896a43009e04deef7b2dac61330"
server_key_version: "kuduclusterkey@0"
I20260927 10:55:43.933837 27205 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:55:43.954602 27220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:43.956507 27205 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 10:55:43.956648 27205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "3fcc3c258494451e838d3275917dedad"
format_stamp: "Formatted at 2026-09-27 10:55:43 on dist-test-slave-tz51"
server_key: "5616c772d546f41787b51fc92ebe3f2c"
server_key_iv: "bed4a896a43009e04deef7b2dac61330"
server_key_version: "kuduclusterkey@0"
I20260927 10:55:43.956816 27205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:43.992532 27205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:43.996413 27205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:43.996608 27205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:43.997326 27205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:43.998402 27205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:43.998479 27205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:43.998543 27205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:43.998579 27205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:44.009466 27205 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:45839
I20260927 10:55:44.010708 27205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:55:44.016952 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 27205
I20260927 10:55:44.017074 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:55:44.017393 14608 external_mini_cluster.cc:1428] Setting key 7c3ced58ff6cde3dad9f35e304941506
I20260927 10:55:44.018316 27333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:45839 every 8 connection(s)
Sep 27 10:55:44 dist-test-slave-tz51 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506543, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:55:44.039635 27065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:44.020498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:36931 (local address 127.14.68.62:46883)
0927 10:55:44.020635 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:44.020640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:44.024064 (+ 3424us) server_negotiation.cc:407] Connection header received
0927 10:55:44.025479 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:44.025482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:44.025539 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:44.025598 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:44.027393 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:44.028221 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:44.031511 (+ 3290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:44.031882 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:44.031970 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:44.035153 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:44.035157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:44.035160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:44.035188 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:44.037080 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:44.037745 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:44.037749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:44.037751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:44.037841 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:44.038286 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:44.038289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:44.038291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:44.038516 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:44.038594 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:44.039330 (+ 736us) server_negotiation.cc:299] Negotiation successful
0927 10:55:44.039466 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:55:44.042033 27336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:44.019309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:46883 (local address 127.14.68.3:36931)
0927 10:55:44.023892 (+ 4583us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:44.023958 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:44.025201 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:44.026111 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:44.026121 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:44.026517 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:44.027206 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:44.027230 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:44.028394 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:44.028397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:44.029796 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:44.029807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:44.032176 (+ 2369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:44.032180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:44.032405 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:44.033149 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:44.033170 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:44.034951 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:44.037227 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:44.037234 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:44.037252 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:44.037623 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:44.037961 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:44.037964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:44.037966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:44.038150 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:44.038675 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:44.038683 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:44.039025 (+ 342us) client_negotiation.cc:769] Sending connection context
0927 10:55:44.040193 (+ 1168us) client_negotiation.cc:240] Negotiation successful
0927 10:55:44.040540 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4319,"thread_start_us":175,"threads_started":1}
I20260927 10:55:44.043651 27334 heartbeater.cc:347] Connected to a master server at 127.14.68.62:46883
I20260927 10:55:44.043951 27334 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:44.044538 27334 heartbeater.cc:511] Master 127.14.68.62:46883 requested a full tablet report, sending...
I20260927 10:55:44.045764 26850 ts_manager.cc:194] Registered new tserver with Master: 3fcc3c258494451e838d3275917dedad (127.14.68.3:45839)
I20260927 10:55:44.046527 26850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:36931
I20260927 10:55:44.055045 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:55:44.055845 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 26931
I20260927 10:55:44.065765 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 27070
I20260927 10:55:44.074429 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 27205
I20260927 10:55:44.082175 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 26818
I20260927 10:55:44.088783 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:46883
--webserver_interface=127.14.68.62
--webserver_port=44023
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:46883
--ranger_config_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.68.20:47007/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.68.20:47007/kms with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:44.197381 27337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:44.197624 27337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:44.197661 27337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:44.201550 27337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:55:44.201596 27337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:44.201615 27337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:44.201630 27337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:55:44.201645 27337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:55:44.206830 27337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.68.20:47007/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:46883
--ranger_config_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:46883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=44023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:44.207769 27337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:44.208686 27337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:44.216830 27342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:44.216835 27345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:44.216889 27337 server_base.cc:1034] running on GCE node
W20260927 10:55:44.216821 27343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:44.217658 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:44.218652 27337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:44.219765 27337 hybrid_clock.cc:648] HybridClock initialized: now 1790506544219753 us; error 25 us; skew 500 ppm
Sep 27 10:55:44 dist-test-slave-tz51 krb5kdc[26374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506544, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:44.222432 27337 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:55:44.223239 27337 webserver.cc:460] Webserver started at http://127.14.68.62:44023/ using document root <none> and password file <none>
I20260927 10:55:44.223822 27337 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:44.223915 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 10:55:44 dist-test-slave-tz51 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506544, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for HTTP/127.14.68.20@KRBTEST.COM
I20260927 10:55:44.287153 27337 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 10:55:44.289870 27353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:44.291298 27337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:55:44.291445 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "4c261ce3483940efbd6fc2503a460757"
format_stamp: "Formatted at 2026-09-27 10:55:42 on dist-test-slave-tz51"
server_key: "a46a4a69a3329d41744217fcec1c9512"
server_key_iv: "6736b046e5e4a2b3ba917cdb39ae023c"
server_key_version: "kuduclusterkey@0"
I20260927 10:55:44.292095 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:44.317895 27337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:44.321759 27337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:44.322201 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:44.329385 27337 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:46883
I20260927 10:55:44.329417 27405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:46883 every 8 connection(s)
I20260927 10:55:44.330435 27337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:55:44.334527 27406 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:55:44.336133 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 27337
I20260927 10:55:44.336154 27406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757: Bootstrap starting.
I20260927 10:55:44.336545 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.68.20:47007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:34977
--local_ip_for_outbound_sockets=127.14.68.1
--tserver_master_addrs=127.14.68.62:46883
--webserver_port=44307
--webserver_interface=127.14.68.1
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.68.20:47007/kms with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:55:44.352700 27406 log.cc:824] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:44.359022 27406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:55:44.359401 27406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757: Bootstrap complete.
I20260927 10:55:44.363435 27406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c261ce3483940efbd6fc2503a460757" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46883 } }
I20260927 10:55:44.363914 27406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c261ce3483940efbd6fc2503a460757, State: Initialized, Role: FOLLOWER
I20260927 10:55:44.364432 27406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c261ce3483940efbd6fc2503a460757" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46883 } }
I20260927 10:55:44.364574 27406 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:55:44.364681 27406 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:55:44.364790 27406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:55:44.366580 27406 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c261ce3483940efbd6fc2503a460757" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46883 } }
I20260927 10:55:44.366899 27406 leader_election.cc:302] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c261ce3483940efbd6fc2503a460757; no voters:
I20260927 10:55:44.367206 27406 leader_election.cc:288] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 10:55:44.367282 27410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 10:55:44.367506 27410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [term 2 LEADER]: Becoming Leader. State: Replica: 4c261ce3483940efbd6fc2503a460757, State: Running, Role: LEADER
I20260927 10:55:44.367862 27410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c261ce3483940efbd6fc2503a460757" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46883 } }
I20260927 10:55:44.368381 27406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:55:44.368785 27411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4c261ce3483940efbd6fc2503a460757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c261ce3483940efbd6fc2503a460757" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46883 } } }
I20260927 10:55:44.368819 27412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c261ce3483940efbd6fc2503a460757. Latest consensus state: current_term: 2 leader_uuid: "4c261ce3483940efbd6fc2503a460757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c261ce3483940efbd6fc2503a460757" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46883 } } }
I20260927 10:55:44.368906 27411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:44.368916 27412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:44.369282 27414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:55:44.372507 27414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:55:44.372990 27406 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 10:55:44.373186 27414 catalog_manager.cc:1259] Loaded cluster ID: ea7dfda805da441fa4b69b4edc24dd01
I20260927 10:55:44.373234 27414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:55:44.378353 27414 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:55:44.379416 27414 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757: Loaded TSK: 0
I20260927 10:55:44.380645 27414 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:55:44.454824 27408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:44.455108 27408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:44.455231 27408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:44.459808 27408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:44.459888 27408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:44.459992 27408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:55:44.465827 27408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.68.20:47007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:34977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=44307
--tserver_master_addrs=127.14.68.62:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:44.467123 27408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:44.468245 27408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:44.478130 27443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:44.478304 27444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:44.478454 27408 server_base.cc:1034] running on GCE node
W20260927 10:55:44.478533 27446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:44.479089 27408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:44.479799 27408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:44.480978 27408 hybrid_clock.cc:648] HybridClock initialized: now 1790506544480945 us; error 43 us; skew 500 ppm
Sep 27 10:55:44 dist-test-slave-tz51 krb5kdc[26374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506544, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:44.484647 27408 init.cc:377] Logged in from keytab as kudu/127.14.68.1@KRBTEST.COM (short username kudu)
I20260927 10:55:44.485755 27408 webserver.cc:460] Webserver started at http://127.14.68.1:44307/ using document root <none> and password file <none>
I20260927 10:55:44.486465 27408 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:44.486555 27408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 10:55:44 dist-test-slave-tz51 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506544, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for HTTP/127.14.68.20@KRBTEST.COM
I20260927 10:55:44.513495 27408 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 10:55:44.516873 27454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:44.518529 27408 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 10:55:44.518675 27408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "eedbc635919f42018cfacbf15ec2c43d"
format_stamp: "Formatted at 2026-09-27 10:55:43 on dist-test-slave-tz51"
server_key: "fb124f475946f32c4689d174a54854d4"
server_key_iv: "27e6b5f03f738c63cec4bb31e14d2611"
server_key_version: "kuduclusterkey@0"
I20260927 10:55:44.519419 27408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:44.561960 27408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:44.565682 27408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:44.565964 27408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:44.566690 27408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:44.567683 27408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:44.567807 27408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:44.567929 27408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:44.568002 27408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:44.578289 27408 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:34977
I20260927 10:55:44.581437 27408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:55:44.584632 27567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:34977 every 8 connection(s)
I20260927 10:55:44.585400 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 27408
I20260927 10:55:44.585858 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.68.20:47007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:38235
--local_ip_for_outbound_sockets=127.14.68.2
--tserver_master_addrs=127.14.68.62:46883
--webserver_port=33967
--webserver_interface=127.14.68.2
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.68.20:47007/kms with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:55:44 dist-test-slave-tz51 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506544, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:55:44.619240 27572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:44.587665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:46883 (local address 127.14.68.1:52051)
0927 10:55:44.588785 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:44.588836 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:44.589814 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:44.599963 (+ 10149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:44.599972 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:44.600246 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:44.600867 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:44.600890 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:44.607557 (+ 6667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:44.607562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:44.609046 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:44.609059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:44.610379 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:44.610384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:44.610586 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:44.611291 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:44.611312 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:44.613187 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:44.615589 (+ 2402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:44.615595 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:44.615608 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:44.615974 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:44.616379 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:44.616382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:44.616384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:44.616548 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:44.617302 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:44.617309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:44.617610 (+ 301us) client_negotiation.cc:769] Sending connection context
0927 10:55:44.617802 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 10:55:44.618152 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"thread_start_us":139,"threads_started":1}
I20260927 10:55:44.621186 27571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:44.587403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:52051 (local address 127.14.68.62:46883)
0927 10:55:44.595730 (+ 8327us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:44.595755 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:44.595805 (+ 50us) server_negotiation.cc:407] Connection header received
0927 10:55:44.596183 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:44.596203 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:44.596684 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:44.597069 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:44.605663 (+ 8594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:44.607328 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:44.609215 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:44.609943 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:44.610134 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:44.613403 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:44.613409 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:44.613426 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:44.613457 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:44.615454 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:44.616122 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:44.616128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:44.616134 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:44.616268 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:44.616662 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:44.616666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:44.616667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:44.617121 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:44.617248 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:44.620031 (+ 2783us) server_negotiation.cc:299] Negotiation successful
0927 10:55:44.620311 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8119,"thread_start_us":128,"threads_started":1}
I20260927 10:55:44.624600 27568 heartbeater.cc:347] Connected to a master server at 127.14.68.62:46883
I20260927 10:55:44.624840 27568 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:44.625434 27568 heartbeater.cc:511] Master 127.14.68.62:46883 requested a full tablet report, sending...
W20260927 10:55:44.627833 27568 heartbeater.cc:644] Failed to heartbeat to 127.14.68.62:46883 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 10:55:44.627941 27568 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:44.628096 27568 heartbeater.cc:511] Master 127.14.68.62:46883 requested a full tablet report, sending...
I20260927 10:55:44.628970 27568 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:44.629142 27568 heartbeater.cc:511] Master 127.14.68.62:46883 requested a full tablet report, sending...
W20260927 10:55:44.629849 27568 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:55:44.771210 27570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:44.771909 27570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:44.772045 27570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:44.777894 27570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:44.777961 27570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:44.778044 27570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:55:44.785356 27570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.68.20:47007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:38235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=33967
--tserver_master_addrs=127.14.68.62:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:44.786772 27570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:44.788102 27570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:44.800244 27578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:44.800367 27580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:44.800639 27570 server_base.cc:1034] running on GCE node
W20260927 10:55:44.802968 27577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:44.803453 27570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:44.804172 27570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:44.805339 27570 hybrid_clock.cc:648] HybridClock initialized: now 1790506544805316 us; error 42 us; skew 500 ppm
Sep 27 10:55:44 dist-test-slave-tz51 krb5kdc[26374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506544, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:44.808674 27570 init.cc:377] Logged in from keytab as kudu/127.14.68.2@KRBTEST.COM (short username kudu)
I20260927 10:55:44.809630 27570 webserver.cc:460] Webserver started at http://127.14.68.2:33967/ using document root <none> and password file <none>
I20260927 10:55:44.810324 27570 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:44.810436 27570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 10:55:44 dist-test-slave-tz51 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506544, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for HTTP/127.14.68.20@KRBTEST.COM
I20260927 10:55:44.843788 27570 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.003s sys 0.001s
I20260927 10:55:44.847103 27588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:44.848738 27570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 10:55:44.848860 27570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "8ad364827623419ea8e6185d45bd7a09"
format_stamp: "Formatted at 2026-09-27 10:55:43 on dist-test-slave-tz51"
server_key: "6bb89a57c7dbe3c3ff289b2a942ffa52"
server_key_iv: "b1532248284ef229742bd473eb3d97e3"
server_key_version: "kuduclusterkey@0"
I20260927 10:55:44.849576 27570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:44.865219 27570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:44.871970 27570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:44.872242 27570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:44.872920 27570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:44.873929 27570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:44.874027 27570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:44.874131 27570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:44.874266 27570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:44.886952 27570 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:38235
I20260927 10:55:44.888285 27570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:55:44.894699 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 27570
I20260927 10:55:44.895203 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.68.20:47007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:45839
--local_ip_for_outbound_sockets=127.14.68.3
--tserver_master_addrs=127.14.68.62:46883
--webserver_port=37617
--webserver_interface=127.14.68.3
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.68.20:47007/kms with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:55:44.890404 27701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:38235 every 8 connection(s)
Sep 27 10:55:44 dist-test-slave-tz51 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506544, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:55:44.913412 27571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:44.893855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:35083 (local address 127.14.68.62:46883)
0927 10:55:44.893976 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:44.893980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:44.897306 (+ 3326us) server_negotiation.cc:407] Connection header received
0927 10:55:44.898492 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:44.898496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:44.898553 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:44.898606 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:44.900495 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:44.901284 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:44.904409 (+ 3125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:44.904761 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:44.904841 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:44.909233 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:44.909237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:44.909239 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:44.909270 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:44.910994 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:44.911595 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:44.911598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:44.911599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:44.911681 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:44.912091 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:44.912094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:44.912096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:44.912302 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:44.912381 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:44.913074 (+ 693us) server_negotiation.cc:299] Negotiation successful
0927 10:55:44.913238 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:55:44.914359 27704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:44.892675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:46883 (local address 127.14.68.2:35083)
0927 10:55:44.897139 (+ 4464us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:44.897196 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:44.898247 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:44.899013 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:44.899022 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:44.899366 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:44.900319 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:44.900339 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:44.901500 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:44.901503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:44.902806 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:44.902816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:44.905043 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:44.905047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:44.905247 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:44.907316 (+ 2069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:44.907336 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:44.909030 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:44.911133 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:44.911139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:44.911156 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:44.911480 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:44.911805 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:44.911808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:44.911810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:44.911976 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:44.912446 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:44.912452 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:44.912771 (+ 319us) client_negotiation.cc:769] Sending connection context
0927 10:55:44.912973 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 10:55:44.913317 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1655,"thread_start_us":143,"threads_started":1}
I20260927 10:55:44.915675 27702 heartbeater.cc:347] Connected to a master server at 127.14.68.62:46883
I20260927 10:55:44.915958 27702 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:44.916541 27702 heartbeater.cc:511] Master 127.14.68.62:46883 requested a full tablet report, sending...
W20260927 10:55:44.918213 27702 heartbeater.cc:644] Failed to heartbeat to 127.14.68.62:46883 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 10:55:44.918337 27702 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:44.918493 27702 heartbeater.cc:511] Master 127.14.68.62:46883 requested a full tablet report, sending...
I20260927 10:55:44.919319 27702 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:44.919472 27702 heartbeater.cc:511] Master 127.14.68.62:46883 requested a full tablet report, sending...
W20260927 10:55:44.920246 27702 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:55:45.066679 27705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:45.067005 27705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:45.067077 27705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:45.073204 27705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:45.073285 27705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:45.073369 27705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:55:45.079794 27705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.68.20:47007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:45839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=37617
--tserver_master_addrs=127.14.68.62:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:45.081056 27705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:45.082198 27705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:45.094437 27711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:45.096736 27712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:45.101399 27714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:45.102102 27705 server_base.cc:1034] running on GCE node
I20260927 10:55:45.102648 27705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:45.103525 27705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:45.104792 27705 hybrid_clock.cc:648] HybridClock initialized: now 1790506545104764 us; error 43 us; skew 500 ppm
Sep 27 10:55:45 dist-test-slave-tz51 krb5kdc[26374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506545, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:45.108712 27705 init.cc:377] Logged in from keytab as kudu/127.14.68.3@KRBTEST.COM (short username kudu)
I20260927 10:55:45.109902 27705 webserver.cc:460] Webserver started at http://127.14.68.3:37617/ using document root <none> and password file <none>
I20260927 10:55:45.110781 27705 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:45.110934 27705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 10:55:45 dist-test-slave-tz51 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506545, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for HTTP/127.14.68.20@KRBTEST.COM
I20260927 10:55:45.136360 27705 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 10:55:45.139202 27722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:45.140720 27705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:55:45.140852 27705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "3fcc3c258494451e838d3275917dedad"
format_stamp: "Formatted at 2026-09-27 10:55:43 on dist-test-slave-tz51"
server_key: "5616c772d546f41787b51fc92ebe3f2c"
server_key_iv: "bed4a896a43009e04deef7b2dac61330"
server_key_version: "kuduclusterkey@0"
I20260927 10:55:45.141589 27705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:45.170269 27705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:45.173717 27705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:45.173969 27705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:45.174714 27705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:45.175733 27705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:45.175827 27705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:45.175910 27705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:45.175961 27705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:45.185556 27705 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:45839
I20260927 10:55:45.186801 27705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:55:45.191969 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 27705
I20260927 10:55:45.195672 27835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:45839 every 8 connection(s)
Sep 27 10:55:45 dist-test-slave-tz51 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506545, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:55:45.205349 27840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:45.192959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49354 (local address 127.14.68.62:46883)
0927 10:55:45.193237 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:45.193242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:45.196239 (+ 2997us) server_negotiation.cc:407] Connection header received
0927 10:55:45.196290 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:45.196293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:45.196338 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:45.196418 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:45.197618 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:45.198484 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:45.199568 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:45.199952 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:45.200061 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:45.201708 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:45.201711 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:45.201714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:45.201759 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:45.203524 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:45.203965 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:45.203969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:45.203971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:45.204052 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:45.204352 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:45.204357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:45.204360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:45.204577 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:45.204773 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:45.205067 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 10:55:45.205183 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":101,"threads_started":1}
I20260927 10:55:45.209048 27571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:45.192000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:49151 (local address 127.14.68.62:46883)
0927 10:55:45.192213 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:45.192218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:45.194933 (+ 2715us) server_negotiation.cc:407] Connection header received
0927 10:55:45.196203 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:45.196206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:45.196278 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:45.196361 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:45.199009 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:45.199820 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:45.201486 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:45.201800 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:45.201913 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:45.204811 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:45.204815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:45.204818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:45.204843 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:45.206671 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:45.207485 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:45.207489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:45.207491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:45.207580 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:45.207976 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:45.207980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:45.207981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:45.208202 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:45.208309 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:45.208800 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 10:55:45.208905 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260927 10:55:45.209900 27838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:45.190928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:46883 (local address 127.14.68.3:49151)
0927 10:55:45.194755 (+ 3827us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:45.194817 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:45.195945 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:45.197815 (+ 1870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:45.197834 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:45.198183 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:45.198847 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:45.198872 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:45.199958 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:45.199965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:45.201351 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:45.201364 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:45.201963 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:45.201967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:45.202178 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:45.202701 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:45.202720 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:45.204649 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:45.206779 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:45.206783 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:45.206794 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:45.207178 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:45.207680 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:45.207686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:45.207690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:45.207873 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:45.208304 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:45.208309 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:45.208532 (+ 223us) client_negotiation.cc:769] Sending connection context
0927 10:55:45.208749 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 10:55:45.209037 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3591,"thread_start_us":141,"threads_started":1}
I20260927 10:55:45.211084 27836 heartbeater.cc:347] Connected to a master server at 127.14.68.62:46883
I20260927 10:55:45.211328 27836 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:45.211910 27836 heartbeater.cc:511] Master 127.14.68.62:46883 requested a full tablet report, sending...
W20260927 10:55:45.213589 27836 heartbeater.cc:644] Failed to heartbeat to 127.14.68.62:46883 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 10:55:45.213703 27836 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:45.213851 27836 heartbeater.cc:511] Master 127.14.68.62:46883 requested a full tablet report, sending...
I20260927 10:55:45.214623 27836 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:45.214783 27836 heartbeater.cc:511] Master 127.14.68.62:46883 requested a full tablet report, sending...
W20260927 10:55:45.216264 27836 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 10:55:45 dist-test-slave-tz51 krb5kdc[26374](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790506545, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:45.630075 27568 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:45.630386 27568 heartbeater.cc:511] Master 127.14.68.62:46883 requested a full tablet report, sending...
I20260927 10:55:45.631601 27365 ts_manager.cc:194] Registered new tserver with Master: eedbc635919f42018cfacbf15ec2c43d (127.14.68.1:34977)
I20260927 10:55:45.633468 27365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:52051
I20260927 10:55:45.920468 27702 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:45.920744 27702 heartbeater.cc:511] Master 127.14.68.62:46883 requested a full tablet report, sending...
I20260927 10:55:45.921509 27365 ts_manager.cc:194] Registered new tserver with Master: 8ad364827623419ea8e6185d45bd7a09 (127.14.68.2:38235)
I20260927 10:55:45.922278 27365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:35083
Sep 27 10:55:45 dist-test-slave-tz51 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506545, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for HTTP/127.14.68.20@KRBTEST.COM
I20260927 10:55:46.216500 27836 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:46.216797 27836 heartbeater.cc:511] Master 127.14.68.62:46883 requested a full tablet report, sending...
I20260927 10:55:46.217619 27365 ts_manager.cc:194] Registered new tserver with Master: 3fcc3c258494451e838d3275917dedad (127.14.68.3:45839)
I20260927 10:55:46.218379 27365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:49151
I20260927 10:55:46.224817 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 10:55:46 dist-test-slave-tz51 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506545, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for HTTP/127.14.68.20@KRBTEST.COM
I20260927 10:55:46.245024 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:46.228422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49364 (local address 127.14.68.62:46883)
0927 10:55:46.231864 (+ 3442us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:46.231869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:46.235551 (+ 3682us) server_negotiation.cc:407] Connection header received
0927 10:55:46.235615 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:46.235618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:46.235670 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:46.235776 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:46.236740 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.237530 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.238709 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.239024 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.239133 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:46.240958 (+ 1825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:46.240962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:46.240964 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:46.240992 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:46.242841 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:46.243301 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:46.243305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:46.243308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:46.243390 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:46.243733 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:46.243737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:46.243739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:46.243944 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:46.244069 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:46.244725 (+ 656us) server_negotiation.cc:299] Negotiation successful
0927 10:55:46.244848 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3357,"thread_start_us":81,"threads_started":1}
I20260927 10:55:46.249845 27365 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:55:46.254891 27365 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:55:46.279129 27866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:46.268724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35184 (local address 127.14.68.2:38235)
0927 10:55:46.272757 (+ 4033us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:46.272763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:46.272787 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:55:46.272864 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:46.272870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:46.273101 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:46.273273 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:46.273813 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.275018 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.277376 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.278200 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.278276 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:46.278429 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:46.278870 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 10:55:46.278971 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3936,"thread_start_us":116,"threads_started":1}
I20260927 10:55:46.279901 27867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:46.267654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:38235 (local address 127.0.0.1:35184)
0927 10:55:46.271624 (+ 3970us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:46.271676 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:46.271832 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:46.273478 (+ 1646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:46.273485 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:46.273511 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:46.273660 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:46.273669 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.275164 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.275168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:46.277232 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:46.277242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.278520 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.278523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:46.278643 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:46.278688 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 10:55:46.279505 (+ 817us) client_negotiation.cc:240] Negotiation successful
0927 10:55:46.279642 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3779,"thread_start_us":101,"threads_started":1}
I20260927 10:55:46.283093 27865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:46.269422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:45839 (local address 127.0.0.1:43568)
0927 10:55:46.277561 (+ 8139us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:46.277576 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:46.277669 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:46.278388 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:46.278391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:46.278409 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:46.278519 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:46.278524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.280158 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.280165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:46.281387 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:46.281396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.282708 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.282712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:46.282809 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:46.282831 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:55:46.282897 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 10:55:46.282956 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8073,"thread_start_us":88,"threads_started":1}
I20260927 10:55:46.283557 27868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:46.269803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43568 (local address 127.14.68.3:45839)
0927 10:55:46.272102 (+ 2299us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:46.272109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:46.277766 (+ 5657us) server_negotiation.cc:407] Connection header received
0927 10:55:46.277846 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:46.277851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:46.278062 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:46.278232 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:46.278642 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.279897 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.281505 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.282299 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.282397 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:46.282646 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:46.283333 (+ 687us) server_negotiation.cc:299] Negotiation successful
0927 10:55:46.283426 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2193,"thread_start_us":73,"threads_started":1}
I20260927 10:55:46.283399 27636 tablet_service.cc:1467] Processing CreateTablet for tablet 4703f6e254c840828017465a7630dace (DEFAULT_TABLE table=test-table [id=b164fd3c70864094bdd294b0a542a9f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:46.284595 27636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4703f6e254c840828017465a7630dace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:46.286810 27870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:46.270646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51080 (local address 127.14.68.1:34977)
0927 10:55:46.272562 (+ 1916us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:46.272569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:46.281137 (+ 8568us) server_negotiation.cc:407] Connection header received
0927 10:55:46.281352 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:46.281362 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:46.281569 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:46.281696 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:46.282070 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.283180 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.284544 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.285302 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.285386 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:46.285512 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:46.286570 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0927 10:55:46.286680 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1808,"thread_start_us":113,"threads_started":1}
I20260927 10:55:46.287194 27867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:46.270440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:34977 (local address 127.0.0.1:51080)
0927 10:55:46.281053 (+ 10613us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:46.281066 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:46.281200 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:46.281873 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:46.281876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:46.281891 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:46.281955 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:46.281960 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.283316 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.283319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:46.284413 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:46.284422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.286323 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.286327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:46.286413 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:46.286435 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:55:46.287014 (+ 579us) client_negotiation.cc:240] Negotiation successful
0927 10:55:46.287087 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10563,"thread_start_us":74,"threads_started":1}
I20260927 10:55:46.286892 27762 tablet_service.cc:1467] Processing CreateTablet for tablet 4703f6e254c840828017465a7630dace (DEFAULT_TABLE table=test-table [id=b164fd3c70864094bdd294b0a542a9f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:46.288101 27762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4703f6e254c840828017465a7630dace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:46.289929 27502 tablet_service.cc:1467] Processing CreateTablet for tablet 4703f6e254c840828017465a7630dace (DEFAULT_TABLE table=test-table [id=b164fd3c70864094bdd294b0a542a9f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:46.290886 27502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4703f6e254c840828017465a7630dace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:46.295748 27873 tablet_bootstrap.cc:492] T 4703f6e254c840828017465a7630dace P 3fcc3c258494451e838d3275917dedad: Bootstrap starting.
I20260927 10:55:46.301992 27873 tablet_bootstrap.cc:654] T 4703f6e254c840828017465a7630dace P 3fcc3c258494451e838d3275917dedad: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:46.302527 27872 tablet_bootstrap.cc:492] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09: Bootstrap starting.
I20260927 10:55:46.303344 27873 log.cc:824] T 4703f6e254c840828017465a7630dace P 3fcc3c258494451e838d3275917dedad: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:46.304097 27874 tablet_bootstrap.cc:492] T 4703f6e254c840828017465a7630dace P eedbc635919f42018cfacbf15ec2c43d: Bootstrap starting.
I20260927 10:55:46.304780 27872 tablet_bootstrap.cc:654] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:46.305658 27872 log.cc:824] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:46.306712 27874 tablet_bootstrap.cc:654] T 4703f6e254c840828017465a7630dace P eedbc635919f42018cfacbf15ec2c43d: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:46.307798 27872 tablet_bootstrap.cc:492] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09: No bootstrap required, opened a new log
I20260927 10:55:46.308007 27872 ts_tablet_manager.cc:1397] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:55:46.308595 27873 tablet_bootstrap.cc:492] T 4703f6e254c840828017465a7630dace P 3fcc3c258494451e838d3275917dedad: No bootstrap required, opened a new log
I20260927 10:55:46.308836 27873 ts_tablet_manager.cc:1397] T 4703f6e254c840828017465a7630dace P 3fcc3c258494451e838d3275917dedad: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260927 10:55:46.312326 27872 raft_consensus.cc:348] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad364827623419ea8e6185d45bd7a09" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 38235 } } peers { permanent_uuid: "3fcc3c258494451e838d3275917dedad" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45839 } } peers { permanent_uuid: "eedbc635919f42018cfacbf15ec2c43d" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 34977 } }
I20260927 10:55:46.312640 27872 raft_consensus.cc:374] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:46.312729 27872 raft_consensus.cc:729] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad364827623419ea8e6185d45bd7a09, State: Initialized, Role: FOLLOWER
I20260927 10:55:46.312836 27873 raft_consensus.cc:348] T 4703f6e254c840828017465a7630dace P 3fcc3c258494451e838d3275917dedad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad364827623419ea8e6185d45bd7a09" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 38235 } } peers { permanent_uuid: "3fcc3c258494451e838d3275917dedad" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45839 } } peers { permanent_uuid: "eedbc635919f42018cfacbf15ec2c43d" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 34977 } }
I20260927 10:55:46.313215 27873 raft_consensus.cc:374] T 4703f6e254c840828017465a7630dace P 3fcc3c258494451e838d3275917dedad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:46.313227 27872 consensus_queue.cc:260] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad364827623419ea8e6185d45bd7a09" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 38235 } } peers { permanent_uuid: "3fcc3c258494451e838d3275917dedad" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45839 } } peers { permanent_uuid: "eedbc635919f42018cfacbf15ec2c43d" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 34977 } }
I20260927 10:55:46.313319 27873 raft_consensus.cc:729] T 4703f6e254c840828017465a7630dace P 3fcc3c258494451e838d3275917dedad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fcc3c258494451e838d3275917dedad, State: Initialized, Role: FOLLOWER
I20260927 10:55:46.313874 27873 consensus_queue.cc:260] T 4703f6e254c840828017465a7630dace P 3fcc3c258494451e838d3275917dedad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad364827623419ea8e6185d45bd7a09" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 38235 } } peers { permanent_uuid: "3fcc3c258494451e838d3275917dedad" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45839 } } peers { permanent_uuid: "eedbc635919f42018cfacbf15ec2c43d" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 34977 } }
I20260927 10:55:46.314316 27872 ts_tablet_manager.cc:1428] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:55:46.314695 27702 heartbeater.cc:503] Master 127.14.68.62:46883 was elected leader, sending a full tablet report...
I20260927 10:55:46.315048 27836 heartbeater.cc:503] Master 127.14.68.62:46883 was elected leader, sending a full tablet report...
I20260927 10:55:46.315791 27874 log.cc:824] T 4703f6e254c840828017465a7630dace P eedbc635919f42018cfacbf15ec2c43d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:46.316748 27873 ts_tablet_manager.cc:1428] T 4703f6e254c840828017465a7630dace P 3fcc3c258494451e838d3275917dedad: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260927 10:55:46.319083 27874 tablet_bootstrap.cc:492] T 4703f6e254c840828017465a7630dace P eedbc635919f42018cfacbf15ec2c43d: No bootstrap required, opened a new log
I20260927 10:55:46.319311 27874 ts_tablet_manager.cc:1397] T 4703f6e254c840828017465a7630dace P eedbc635919f42018cfacbf15ec2c43d: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260927 10:55:46.323567 27874 raft_consensus.cc:348] T 4703f6e254c840828017465a7630dace P eedbc635919f42018cfacbf15ec2c43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad364827623419ea8e6185d45bd7a09" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 38235 } } peers { permanent_uuid: "3fcc3c258494451e838d3275917dedad" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45839 } } peers { permanent_uuid: "eedbc635919f42018cfacbf15ec2c43d" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 34977 } }
I20260927 10:55:46.323868 27874 raft_consensus.cc:374] T 4703f6e254c840828017465a7630dace P eedbc635919f42018cfacbf15ec2c43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:46.323930 27874 raft_consensus.cc:729] T 4703f6e254c840828017465a7630dace P eedbc635919f42018cfacbf15ec2c43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eedbc635919f42018cfacbf15ec2c43d, State: Initialized, Role: FOLLOWER
I20260927 10:55:46.324373 27874 consensus_queue.cc:260] T 4703f6e254c840828017465a7630dace P eedbc635919f42018cfacbf15ec2c43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad364827623419ea8e6185d45bd7a09" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 38235 } } peers { permanent_uuid: "3fcc3c258494451e838d3275917dedad" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45839 } } peers { permanent_uuid: "eedbc635919f42018cfacbf15ec2c43d" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 34977 } }
I20260927 10:55:46.325330 27874 ts_tablet_manager.cc:1428] T 4703f6e254c840828017465a7630dace P eedbc635919f42018cfacbf15ec2c43d: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260927 10:55:46.325330 27568 heartbeater.cc:503] Master 127.14.68.62:46883 was elected leader, sending a full tablet report...
W20260927 10:55:46.337373 27569 tablet.cc:2369] T 4703f6e254c840828017465a7630dace P eedbc635919f42018cfacbf15ec2c43d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:55:46.389743 27877 raft_consensus.cc:482] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:55:46.390015 27877 raft_consensus.cc:504] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad364827623419ea8e6185d45bd7a09" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 38235 } } peers { permanent_uuid: "3fcc3c258494451e838d3275917dedad" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45839 } } peers { permanent_uuid: "eedbc635919f42018cfacbf15ec2c43d" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 34977 } }
I20260927 10:55:46.391476 27877 leader_election.cc:288] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fcc3c258494451e838d3275917dedad (127.14.68.3:45839), eedbc635919f42018cfacbf15ec2c43d (127.14.68.1:34977)
W20260927 10:55:46.395511 27703 tablet.cc:2369] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:55:46.398808 27868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:46.392317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:45991 (local address 127.14.68.3:45839)
0927 10:55:46.392467 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:46.392472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:46.393021 (+ 549us) server_negotiation.cc:407] Connection header received
0927 10:55:46.393363 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:46.393367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:46.393419 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:46.393467 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:46.393935 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.394746 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.397065 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.397693 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.397801 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:46.397890 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:46.398601 (+ 711us) server_negotiation.cc:299] Negotiation successful
0927 10:55:46.398680 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:55:46.399240 27882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:46.391883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:45839 (local address 127.14.68.2:45991)
0927 10:55:46.392921 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:46.392939 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:46.393223 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:46.393638 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:46.393641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:46.393659 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:46.393800 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:46.393807 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.394891 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.394894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:46.396903 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:46.396910 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.398309 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.398313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:46.398429 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:46.398453 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 10:55:46.399002 (+ 549us) client_negotiation.cc:240] Negotiation successful
0927 10:55:46.399084 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"thread_start_us":117,"threads_started":1}
I20260927 10:55:46.399905 27790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4703f6e254c840828017465a7630dace" candidate_uuid: "8ad364827623419ea8e6185d45bd7a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fcc3c258494451e838d3275917dedad" is_pre_election: true
I20260927 10:55:46.400221 27790 raft_consensus.cc:2454] T 4703f6e254c840828017465a7630dace P 3fcc3c258494451e838d3275917dedad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad364827623419ea8e6185d45bd7a09 in term 0.
I20260927 10:55:46.400731 27589 leader_election.cc:302] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fcc3c258494451e838d3275917dedad, 8ad364827623419ea8e6185d45bd7a09; no voters:
I20260927 10:55:46.401047 27877 raft_consensus.cc:2781] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:55:46.401149 27877 raft_consensus.cc:482] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:55:46.401203 27877 raft_consensus.cc:3037] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:46.401978 27870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:46.392987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:43511 (local address 127.14.68.1:34977)
0927 10:55:46.393141 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:46.393146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:46.396741 (+ 3595us) server_negotiation.cc:407] Connection header received
0927 10:55:46.396928 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:46.396934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:46.396998 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:46.397053 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:46.397373 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.398152 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.400691 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.401276 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.401379 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:46.401437 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:46.401783 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 10:55:46.401861 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 10:55:46.402334 27881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:46.392563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:34977 (local address 127.14.68.2:43511)
0927 10:55:46.396605 (+ 4042us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:46.396621 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:46.396804 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:46.397165 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:46.397167 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:46.397179 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:46.397253 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:46.397256 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.399096 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.399099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:46.400560 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:46.400569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.401497 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.401500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:46.401615 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:46.401656 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 10:55:46.402124 (+ 468us) client_negotiation.cc:240] Negotiation successful
0927 10:55:46.402199 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3977,"thread_start_us":137,"threads_started":1}
I20260927 10:55:46.402558 27877 raft_consensus.cc:504] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad364827623419ea8e6185d45bd7a09" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 38235 } } peers { permanent_uuid: "3fcc3c258494451e838d3275917dedad" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45839 } } peers { permanent_uuid: "eedbc635919f42018cfacbf15ec2c43d" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 34977 } }
I20260927 10:55:46.402907 27877 leader_election.cc:288] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09 [CANDIDATE]: Term 1 election: Requested vote from peers 3fcc3c258494451e838d3275917dedad (127.14.68.3:45839), eedbc635919f42018cfacbf15ec2c43d (127.14.68.1:34977)
I20260927 10:55:46.403868 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4703f6e254c840828017465a7630dace" candidate_uuid: "8ad364827623419ea8e6185d45bd7a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eedbc635919f42018cfacbf15ec2c43d"
I20260927 10:55:46.404096 27519 raft_consensus.cc:3037] T 4703f6e254c840828017465a7630dace P eedbc635919f42018cfacbf15ec2c43d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:46.405480 27519 raft_consensus.cc:2454] T 4703f6e254c840828017465a7630dace P eedbc635919f42018cfacbf15ec2c43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad364827623419ea8e6185d45bd7a09 in term 1.
I20260927 10:55:46.405766 27506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4703f6e254c840828017465a7630dace" candidate_uuid: "8ad364827623419ea8e6185d45bd7a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eedbc635919f42018cfacbf15ec2c43d" is_pre_election: true
I20260927 10:55:46.405896 27506 raft_consensus.cc:2362] T 4703f6e254c840828017465a7630dace P eedbc635919f42018cfacbf15ec2c43d [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 8ad364827623419ea8e6185d45bd7a09 in term 1. Re-sending same reply.
I20260927 10:55:46.406276 27790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4703f6e254c840828017465a7630dace" candidate_uuid: "8ad364827623419ea8e6185d45bd7a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fcc3c258494451e838d3275917dedad"
I20260927 10:55:46.406407 27790 raft_consensus.cc:3037] T 4703f6e254c840828017465a7630dace P 3fcc3c258494451e838d3275917dedad [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:46.407646 27790 raft_consensus.cc:2454] T 4703f6e254c840828017465a7630dace P 3fcc3c258494451e838d3275917dedad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad364827623419ea8e6185d45bd7a09 in term 1.
I20260927 10:55:46.407982 27589 leader_election.cc:302] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fcc3c258494451e838d3275917dedad, 8ad364827623419ea8e6185d45bd7a09; no voters:
I20260927 10:55:46.408706 27877 raft_consensus.cc:2781] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:55:46.408926 27877 raft_consensus.cc:686] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad364827623419ea8e6185d45bd7a09, State: Running, Role: LEADER
I20260927 10:55:46.409286 27877 consensus_queue.cc:237] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad364827623419ea8e6185d45bd7a09" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 38235 } } peers { permanent_uuid: "3fcc3c258494451e838d3275917dedad" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45839 } } peers { permanent_uuid: "eedbc635919f42018cfacbf15ec2c43d" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 34977 } }
I20260927 10:55:46.412647 27370 catalog_manager.cc:5495] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad364827623419ea8e6185d45bd7a09 (127.14.68.2). New cstate: current_term: 1 leader_uuid: "8ad364827623419ea8e6185d45bd7a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad364827623419ea8e6185d45bd7a09" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 38235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fcc3c258494451e838d3275917dedad" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eedbc635919f42018cfacbf15ec2c43d" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 34977 } health_report { overall_health: UNKNOWN } } }
I20260927 10:55:46.439114 27866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:46.433868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35188 (local address 127.14.68.2:38235)
0927 10:55:46.434015 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:46.434020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:46.434232 (+ 212us) server_negotiation.cc:407] Connection header received
0927 10:55:46.434307 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:46.434310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:46.434375 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:46.434470 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:55:46.434831 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.435606 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.436926 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:46.437271 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:46.437370 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:46.437774 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:55:46.438345 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:55:46.438460 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:46.438880 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 10:55:46.438942 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:55:46.446599 27790 raft_consensus.cc:1261] T 4703f6e254c840828017465a7630dace P 3fcc3c258494451e838d3275917dedad [term 1 FOLLOWER]: Refusing update from remote peer 8ad364827623419ea8e6185d45bd7a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:55:46.447518 27506 raft_consensus.cc:1261] T 4703f6e254c840828017465a7630dace P eedbc635919f42018cfacbf15ec2c43d [term 1 FOLLOWER]: Refusing update from remote peer 8ad364827623419ea8e6185d45bd7a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 10:55:46.448091 27837 tablet.cc:2369] T 4703f6e254c840828017465a7630dace P 3fcc3c258494451e838d3275917dedad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:55:46.449183 27877 consensus_queue.cc:1035] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eedbc635919f42018cfacbf15ec2c43d" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 34977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:46.449599 27877 consensus_queue.cc:1035] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fcc3c258494451e838d3275917dedad" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:46.463553 27890 mvcc.cc:205] Tried to move back new op lower bound from 7333914814234759168 to 7333914814095667200. Current Snapshot: MvccSnapshot[applied={T|T < 7333914814234759168}]
I20260927 10:55:46.465250 27886 mvcc.cc:205] Tried to move back new op lower bound from 7333914814234759168 to 7333914814095667200. Current Snapshot: MvccSnapshot[applied={T|T < 7333914814234759168}]
I20260927 10:55:46.468724 27888 mvcc.cc:205] Tried to move back new op lower bound from 7333914814234759168 to 7333914814095667200. Current Snapshot: MvccSnapshot[applied={T|T < 7333914814234759168}]
I20260927 10:55:46.480974 27369 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:55:46.481187 27369 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:55:46.484046 27369 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c261ce3483940efbd6fc2503a460757: Sending DeleteTablet for 3 replicas of tablet 4703f6e254c840828017465a7630dace
I20260927 10:55:46.485337 27636 tablet_service.cc:1514] Processing DeleteTablet for tablet 4703f6e254c840828017465a7630dace with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:46 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:35184
I20260927 10:55:46.485869 27762 tablet_service.cc:1514] Processing DeleteTablet for tablet 4703f6e254c840828017465a7630dace with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:46 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:43568
I20260927 10:55:46.486717 27502 tablet_service.cc:1514] Processing DeleteTablet for tablet 4703f6e254c840828017465a7630dace with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:46 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:51080
I20260927 10:55:46.488161 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 27408
I20260927 10:55:46.499449 27896 tablet_replica.cc:330] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09: stopping tablet replica
I20260927 10:55:46.499882 27896 raft_consensus.cc:2229] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:55:46.500494 27896 raft_consensus.cc:2258] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:46.500757 27897 tablet_replica.cc:330] T 4703f6e254c840828017465a7630dace P 3fcc3c258494451e838d3275917dedad: stopping tablet replica
I20260927 10:55:46.506603 27897 raft_consensus.cc:2229] T 4703f6e254c840828017465a7630dace P 3fcc3c258494451e838d3275917dedad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:55:46.506965 27897 raft_consensus.cc:2258] T 4703f6e254c840828017465a7630dace P 3fcc3c258494451e838d3275917dedad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:46.508665 27897 ts_tablet_manager.cc:1905] T 4703f6e254c840828017465a7630dace P 3fcc3c258494451e838d3275917dedad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:46.511402 27897 ts_tablet_manager.cc:1918] T 4703f6e254c840828017465a7630dace P 3fcc3c258494451e838d3275917dedad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:55:46.511498 27897 log.cc:1196] T 4703f6e254c840828017465a7630dace P 3fcc3c258494451e838d3275917dedad: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/4703f6e254c840828017465a7630dace
I20260927 10:55:46.511824 27897 ts_tablet_manager.cc:1939] T 4703f6e254c840828017465a7630dace P 3fcc3c258494451e838d3275917dedad: Deleting consensus metadata
I20260927 10:55:46.513190 27354 catalog_manager.cc:4841] TS 3fcc3c258494451e838d3275917dedad (127.14.68.3:45839): tablet 4703f6e254c840828017465a7630dace (table test-table [id=b164fd3c70864094bdd294b0a542a9f3]) successfully deleted
I20260927 10:55:46.513665 27896 ts_tablet_manager.cc:1905] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:46.516528 27896 ts_tablet_manager.cc:1918] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:55:46.516640 27896 log.cc:1196] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/4703f6e254c840828017465a7630dace
I20260927 10:55:46.516992 27896 ts_tablet_manager.cc:1939] T 4703f6e254c840828017465a7630dace P 8ad364827623419ea8e6185d45bd7a09: Deleting consensus metadata
I20260927 10:55:46.518337 27355 catalog_manager.cc:4841] TS 8ad364827623419ea8e6185d45bd7a09 (127.14.68.2:38235): tablet 4703f6e254c840828017465a7630dace (table test-table [id=b164fd3c70864094bdd294b0a542a9f3]) successfully deleted
W20260927 10:55:46.518707 27357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.68.1:34977 (error 108)
I20260927 10:55:46.520215 27865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:46.519926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:34977 (local address 127.0.0.1:51086)
0927 10:55:46.520041 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:46.520121 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.1:34977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 10:55:46.520210 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 27570
W20260927 10:55:46.520398 27357 catalog_manager.cc:4584] TS eedbc635919f42018cfacbf15ec2c43d (127.14.68.1:34977): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4703f6e254c840828017465a7630dace: Network error: Client connection negotiation failed: client connection to 127.14.68.1:34977: connect: Connection refused (error 111)
I20260927 10:55:46.527570 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 27705
I20260927 10:55:46.533263 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 27337
I20260927 10:55:46.539472 14608 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 10:55:46.873992 14608 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 10:55:46.874100 14608 mini_ranger.cc:66] Stopping Ranger...
I20260927 10:55:47.218788 14608 mini_ranger.cc:68] Stopped Ranger
2026-09-27 10:55:47.218 UTC [26404] LOG: received smart shutdown request
2026-09-27 10:55:47.223 UTC [26404] LOG: background worker "logical replication launcher" (PID 26413) exited with exit code 1
2026-09-27 10:55:47.223 UTC [26408] LOG: shutting down
2026-09-27 10:55:47.399 UTC [26404] LOG: database system is shut down
2026-09-27T10:55:47Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49032 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:55:47 dist-test-slave-tz51 krb5kdc[27915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:55:47 dist-test-slave-tz51 krb5kdc[27915](info): set up 2 sockets
Sep 27 10:55:47 dist-test-slave-tz51 krb5kdc[27915](info): commencing operation
krb5kdc: starting...
W20260927 10:55:48.539829 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:55:48 dist-test-slave-tz51 krb5kdc[27915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:55:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:55:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:55:48.658602 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:45875
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:34511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:45875,127.14.68.61:37097,127.14.68.60:46131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:48.769392 27928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:48.769672 27928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:48.769745 27928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:48.773766 27928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:55:48.773846 27928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:48.773885 27928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:48.773945 27928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:55:48.773980 27928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:55:48.779026 27928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:34511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:45875,127.14.68.61:37097,127.14.68.60:46131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:45875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:48.780061 27928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:48.781006 27928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:48.789247 27936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:48.789247 27934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:48.789395 27928 server_base.cc:1034] running on GCE node
W20260927 10:55:48.789243 27933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:48.790040 27928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:48.791096 27928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:48.792299 27928 hybrid_clock.cc:648] HybridClock initialized: now 1790506548792288 us; error 24 us; skew 500 ppm
Sep 27 10:55:48 dist-test-slave-tz51 krb5kdc[27915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506548, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:48.795045 27928 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:55:48.795946 27928 webserver.cc:460] Webserver started at http://127.14.68.62:44587/ using document root <none> and password file <none>
I20260927 10:55:48.796419 27928 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:48.796473 27928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:48.796623 27928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:48.798470 27928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "52e47af1c5134896b489f95003d4cafd"
format_stamp: "Formatted at 2026-09-27 10:55:48 on dist-test-slave-tz51"
server_key: "d5b1e9314cb66e4b5be2f0020b0b2f24"
server_key_iv: "500789b38e96985febac32f77461748e"
server_key_version: "encryptionkey@0"
I20260927 10:55:48.798969 27928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "52e47af1c5134896b489f95003d4cafd"
format_stamp: "Formatted at 2026-09-27 10:55:48 on dist-test-slave-tz51"
server_key: "d5b1e9314cb66e4b5be2f0020b0b2f24"
server_key_iv: "500789b38e96985febac32f77461748e"
server_key_version: "encryptionkey@0"
I20260927 10:55:48.802423 27928 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:55:48.804677 27943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:48.805810 27928 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:55:48.805912 27928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "52e47af1c5134896b489f95003d4cafd"
format_stamp: "Formatted at 2026-09-27 10:55:48 on dist-test-slave-tz51"
server_key: "d5b1e9314cb66e4b5be2f0020b0b2f24"
server_key_iv: "500789b38e96985febac32f77461748e"
server_key_version: "encryptionkey@0"
I20260927 10:55:48.805989 27928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:48.833878 27928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:48.837267 27928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:48.837422 27928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:48.844579 27928 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:45875
I20260927 10:55:48.844686 27995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:45875 every 8 connection(s)
I20260927 10:55:48.845579 27928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:55:48.848392 27996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:48.851974 27996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45875 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:55:48.854655 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 27928
I20260927 10:55:48.854808 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:55:48.855114 14608 external_mini_cluster.cc:1428] Setting key ff9bc31b669c446171c8da282121050e
Sep 27 10:55:48 dist-test-slave-tz51 krb5kdc[27915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506548, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.68.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.61@KRBTEST.COM" created.
I20260927 10:55:48.866555 27998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:48.854159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.14.68.62:45875)
0927 10:55:48.854630 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:48.854650 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:48.855009 (+ 359us) server_negotiation.cc:407] Connection header received
0927 10:55:48.856028 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:48.856032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:48.856204 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:48.856391 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:48.857205 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:48.858051 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:48.859007 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:48.859368 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:48.859535 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:48.861859 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:48.861872 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:48.861882 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:48.861902 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:48.863457 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:48.864011 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:48.864016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:48.864018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:48.864144 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:48.864451 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:48.864454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:48.864456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:48.864840 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:48.864974 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:48.865318 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 10:55:48.865556 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":313,"thread_start_us":140,"threads_started":1}
I20260927 10:55:48.866556 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:48.854159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:45875 (local address 127.0.0.1:56720)
0927 10:55:48.854745 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:48.854791 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:48.855680 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:48.856386 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:48.856391 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:48.856809 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:48.857042 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:48.857055 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:48.858184 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:48.858192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:48.858886 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:48.858894 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:48.859518 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:48.859522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:48.859697 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:48.860326 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:48.860343 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:48.861722 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:48.863590 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:48.863598 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:48.863606 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:48.863908 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:48.864243 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:48.864246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:48.864248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:48.864357 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:48.864947 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:48.864953 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:48.865091 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 10:55:48.865247 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 10:55:48.865612 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":112,"spinlock_wait_cycles":48384,"thread_start_us":140,"threads_started":1}
I20260927 10:55:48.869019 27996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:55:48.869729 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:48.869494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:37097 (local address 127.0.0.1:44600)
0927 10:55:48.869600 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:48.869676 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.61:37097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 10:55:48.869989 27945 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.68.61:37097: connect: Connection refused (error 111)
I20260927 10:55:48.870761 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:48.870576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:37097 (local address 127.0.0.1:44616)
0927 10:55:48.870674 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:48.870715 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.61:37097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 10:55:48.870970 27996 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.61:37097: Network error: Client connection negotiation failed: client connection to 127.14.68.61:37097: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.61.keytab.
Entry for principal kudu/127.14.68.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.61.keytab.
I20260927 10:55:48.880425 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.61:37097
--webserver_interface=127.14.68.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.61.keytab
--principal=kudu/127.14.68.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:34511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:45875,127.14.68.61:37097,127.14.68.60:46131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:55:48.920243 27996 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } attempt: 1
I20260927 10:55:48.921232 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:48.920890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:37097 (local address 127.0.0.1:44626)
0927 10:55:48.921060 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:48.921154 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.61:37097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 10:55:48.922034 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:48.921762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:37097 (local address 127.0.0.1:44628)
0927 10:55:48.921873 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:48.921946 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.61:37097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 10:55:48.922185 27996 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.61:37097: Network error: Client connection negotiation failed: client connection to 127.14.68.61:37097: connect: Connection refused (error 111)
W20260927 10:55:48.988991 28003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:48.989275 28003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:48.989346 28003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 10:55:48.990346 27996 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } attempt: 2
I20260927 10:55:48.991184 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:48.990842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:37097 (local address 127.0.0.1:44630)
0927 10:55:48.990988 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:48.991061 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.61:37097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 10:55:48.991911 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:48.991682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:37097 (local address 127.0.0.1:44634)
0927 10:55:48.991815 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:48.991866 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.61:37097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 10:55:48.992033 27996 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.61:37097: Network error: Client connection negotiation failed: client connection to 127.14.68.61:37097: connect: Connection refused (error 111)
W20260927 10:55:48.993600 28003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:55:48.993671 28003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:48.993714 28003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:48.993767 28003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:55:48.993804 28003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:55:48.998946 28003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:34511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:45875,127.14.68.61:37097,127.14.68.60:46131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.61.keytab
--principal=kudu/127.14.68.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.61:37097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.68.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:49.000034 28003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:49.001019 28003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:49.009526 28008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:49.009536 28009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:49.009536 28011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:49.009701 28003 server_base.cc:1034] running on GCE node
I20260927 10:55:49.010320 28003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:49.011404 28003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:49.012580 28003 hybrid_clock.cc:648] HybridClock initialized: now 1790506549012537 us; error 57 us; skew 500 ppm
Sep 27 10:55:49 dist-test-slave-tz51 krb5kdc[27915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506549, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:49.015317 28003 init.cc:377] Logged in from keytab as kudu/127.14.68.61@KRBTEST.COM (short username kudu)
I20260927 10:55:49.016248 28003 webserver.cc:460] Webserver started at http://127.14.68.61:37741/ using document root <none> and password file <none>
I20260927 10:55:49.016819 28003 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:49.016909 28003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:49.017134 28003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:49.019114 28003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-1/data/instance:
uuid: "4c7e2fb24a894b158870888bd4ee5a12"
format_stamp: "Formatted at 2026-09-27 10:55:49 on dist-test-slave-tz51"
server_key: "915ad16976c41717740144386625ca68"
server_key_iv: "ff24e832c804de374639dfd994384ca0"
server_key_version: "encryptionkey@0"
I20260927 10:55:49.019671 28003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-1/wal/instance:
uuid: "4c7e2fb24a894b158870888bd4ee5a12"
format_stamp: "Formatted at 2026-09-27 10:55:49 on dist-test-slave-tz51"
server_key: "915ad16976c41717740144386625ca68"
server_key_iv: "ff24e832c804de374639dfd994384ca0"
server_key_version: "encryptionkey@0"
I20260927 10:55:49.023324 28003 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:55:49.025744 28018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:49.026894 28003 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:49.027029 28003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-1/wal
uuid: "4c7e2fb24a894b158870888bd4ee5a12"
format_stamp: "Formatted at 2026-09-27 10:55:49 on dist-test-slave-tz51"
server_key: "915ad16976c41717740144386625ca68"
server_key_iv: "ff24e832c804de374639dfd994384ca0"
server_key_version: "encryptionkey@0"
I20260927 10:55:49.027135 28003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:49.043534 28003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:49.046557 28003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:49.046763 28003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:49.053946 28003 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.61:37097
I20260927 10:55:49.053969 28070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.61:37097 every 8 connection(s)
I20260927 10:55:49.054874 28003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-1/data/info.pb
I20260927 10:55:49.055951 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 28003
I20260927 10:55:49.056075 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-1/wal/instance
I20260927 10:55:49.056349 14608 external_mini_cluster.cc:1428] Setting key bb70fb435cee3d3d5e2b6e124c0fe042
I20260927 10:55:49.058153 28071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:49.062388 28071 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45875 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.68.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.60@KRBTEST.COM" created.
Sep 27 10:55:49 dist-test-slave-tz51 krb5kdc[27915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506549, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.61@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:55:49.079113 27998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.064367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.14.68.62:45875)
0927 10:55:49.064546 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:49.064551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:49.065367 (+ 816us) server_negotiation.cc:407] Connection header received
0927 10:55:49.066582 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:49.066589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:49.066666 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:49.066774 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:49.068362 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.069309 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.071081 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.071340 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.071436 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:49.074513 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:49.074520 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:49.074523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:49.074558 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:49.076247 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.076781 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.076789 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.076791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.076867 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.077229 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.077234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.077240 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.077539 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:49.077657 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:49.078820 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0927 10:55:49.078952 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 10:55:49.079591 28074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.064763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:45875 (local address 127.0.0.1:56730)
0927 10:55:49.065205 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:49.065250 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:49.066346 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:49.066966 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:49.066976 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:49.067451 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:49.068176 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:49.068202 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.069444 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.069447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:49.070950 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:49.070963 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.071509 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.071516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:49.071786 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:49.072486 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:49.072514 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:49.074325 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:49.076378 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:49.076383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:49.076396 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:49.076673 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:49.076975 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:49.076978 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:49.076979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:49.077135 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:49.077655 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:49.077661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:49.077904 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 10:55:49.078151 (+ 247us) client_negotiation.cc:240] Negotiation successful
0927 10:55:49.078562 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":119,"threads_started":1}
I20260927 10:55:49.081118 28071 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.60.keytab.
Entry for principal kudu/127.14.68.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.60.keytab.
I20260927 10:55:49.082962 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.60:46131
--webserver_interface=127.14.68.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.60.keytab
--principal=kudu/127.14.68.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:34511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:45875,127.14.68.61:37097,127.14.68.60:46131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:55:49.083250 27996 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } attempt: 3
Sep 27 10:55:49 dist-test-slave-tz51 krb5kdc[27915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506549, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.61@KRBTEST.COM for kudu/127.14.68.61@KRBTEST.COM
Sep 27 10:55:49 dist-test-slave-tz51 krb5kdc[27915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506548, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for kudu/127.14.68.61@KRBTEST.COM
I20260927 10:55:49.092307 28074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.081653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:37097 (local address 127.0.0.1:44648)
0927 10:55:49.081775 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:49.081790 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:49.081871 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:49.082511 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:49.082513 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:49.082679 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:49.082755 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:49.082760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.084312 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.084318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:49.085276 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:49.085288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.086113 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.086117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:49.086228 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:49.087055 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:49.087065 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:49.088686 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:49.090547 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:49.090551 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:49.090553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:49.090770 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:49.091113 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:49.091115 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:49.091117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:49.091190 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:49.091786 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:49.091790 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:49.091876 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 10:55:49.091996 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 10:55:49.092127 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":48}
I20260927 10:55:49.092353 28076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.081738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44648 (local address 127.14.68.61:37097)
0927 10:55:49.082120 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:49.082134 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:49.082160 (+ 26us) server_negotiation.cc:407] Connection header received
0927 10:55:49.082215 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:49.082222 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:49.082393 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:49.082533 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:49.082859 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.084207 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.085395 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.085978 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.086119 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:49.088801 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:49.088812 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:49.088828 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:49.088847 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:49.090425 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.090891 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.090900 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.090903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.091001 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.091296 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.091301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.091307 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.091591 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:49.091813 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:49.092076 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 10:55:49.092213 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":149,"threads_started":1}
I20260927 10:55:49.094323 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.083618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:37097 (local address 127.0.0.1:44654)
0927 10:55:49.083828 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:49.083844 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:49.083923 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:49.084274 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:49.084280 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:49.084460 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:49.084570 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:49.084577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.085571 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.085573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:49.086149 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:49.086158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.086740 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.086744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:49.086850 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:49.087286 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:49.087301 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:49.089649 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:49.092813 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:49.092816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:49.092818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:49.093045 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:49.093373 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:49.093376 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:49.093378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:49.093469 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:49.093807 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:49.093810 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:49.093900 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 10:55:49.094004 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 10:55:49.094162 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260927 10:55:49.094357 28077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.083737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44654 (local address 127.14.68.61:37097)
0927 10:55:49.084030 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:49.084038 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:49.084050 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:55:49.084081 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:49.084083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:49.084118 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:49.084680 (+ 562us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:49.084760 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.085485 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.086278 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.086556 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.086736 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:49.090424 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:49.090432 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:49.090438 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:49.090468 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:49.092525 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.093152 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.093159 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.093163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.093256 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.093553 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.093557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.093559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.093713 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:49.093813 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:49.094077 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 10:55:49.094195 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":234,"thread_start_us":133,"threads_started":1}
I20260927 10:55:49.094705 28071 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:55:49.095458 28074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.095172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:46131 (local address 127.0.0.1:49584)
0927 10:55:49.095288 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:49.095374 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 10:55:49.095403 27996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } has no permanent_uuid. Determining permanent_uuid...
W20260927 10:55:49.095850 28021 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.68.60:46131: connect: Connection refused (error 111)
I20260927 10:55:49.096196 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.095933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:46131 (local address 127.0.0.1:49586)
0927 10:55:49.096069 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:49.096130 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 10:55:49.096839 28074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.096612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:46131 (local address 127.0.0.1:49604)
0927 10:55:49.096735 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:49.096783 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 10:55:49.096875 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.096619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:46131 (local address 127.0.0.1:49588)
0927 10:55:49.096792 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:49.096829 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260927 10:55:49.097043 27996 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.60:46131: Network error: Client connection negotiation failed: client connection to 127.14.68.60:46131: connect: Connection refused (error 111)
W20260927 10:55:49.097011 28071 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.60:46131: Network error: Client connection negotiation failed: client connection to 127.14.68.60:46131: connect: Connection refused (error 111)
I20260927 10:55:49.128209 27996 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } attempt: 1
I20260927 10:55:49.128998 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.128729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:46131 (local address 127.0.0.1:49606)
0927 10:55:49.128862 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:49.128927 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 10:55:49.129733 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.129527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:46131 (local address 127.0.0.1:49614)
0927 10:55:49.129638 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:49.129687 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 10:55:49.129879 27996 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.60:46131: Network error: Client connection negotiation failed: client connection to 127.14.68.60:46131: connect: Connection refused (error 111)
I20260927 10:55:49.146257 28071 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } attempt: 1
I20260927 10:55:49.147091 28074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.146803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:46131 (local address 127.0.0.1:49616)
0927 10:55:49.146950 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:49.147010 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 10:55:49.147857 28074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.147608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:46131 (local address 127.0.0.1:49620)
0927 10:55:49.147764 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:49.147807 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 10:55:49.147984 28071 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.60:46131: Network error: Client connection negotiation failed: client connection to 127.14.68.60:46131: connect: Connection refused (error 111)
W20260927 10:55:49.195400 28078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:49.195681 28078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:49.195773 28078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:49.199842 28078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:55:49.199913 28078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:49.199949 28078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:49.199982 28078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:55:49.200017 28078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:55:49.205035 27996 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } attempt: 2
I20260927 10:55:49.205090 28078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:34511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:45875,127.14.68.61:37097,127.14.68.60:46131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.60.keytab
--principal=kudu/127.14.68.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.60:46131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.68.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:49.205981 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.205611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:46131 (local address 127.0.0.1:49624)
0927 10:55:49.205808 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:49.205884 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 10:55:49.206321 28078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:49.206825 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.206583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:46131 (local address 127.0.0.1:49632)
0927 10:55:49.206708 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:49.206762 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 10:55:49.206976 27996 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.60:46131: Network error: Client connection negotiation failed: client connection to 127.14.68.60:46131: connect: Connection refused (error 111)
I20260927 10:55:49.207476 28078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:49.215624 28085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:49.215672 28087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:49.215893 28078 server_base.cc:1034] running on GCE node
W20260927 10:55:49.215644 28084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:49.216147 28071 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } attempt: 2
I20260927 10:55:49.216826 28074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.216562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:46131 (local address 127.0.0.1:49640)
0927 10:55:49.216706 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:49.216756 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 10:55:49.216989 28078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:49.217521 28074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.217343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:46131 (local address 127.0.0.1:49656)
0927 10:55:49.217435 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:49.217475 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 10:55:49.217733 28071 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.60:46131: Network error: Client connection negotiation failed: client connection to 127.14.68.60:46131: connect: Connection refused (error 111)
I20260927 10:55:49.218251 28078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:49.219447 28078 hybrid_clock.cc:648] HybridClock initialized: now 1790506549219424 us; error 32 us; skew 500 ppm
Sep 27 10:55:49 dist-test-slave-tz51 krb5kdc[27915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506549, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:49.222267 28078 init.cc:377] Logged in from keytab as kudu/127.14.68.60@KRBTEST.COM (short username kudu)
I20260927 10:55:49.223160 28078 webserver.cc:460] Webserver started at http://127.14.68.60:41911/ using document root <none> and password file <none>
I20260927 10:55:49.223737 28078 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:49.223834 28078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:49.224061 28078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:49.226018 28078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-2/data/instance:
uuid: "bcda15a0034242a1b3039a562bbf84d7"
format_stamp: "Formatted at 2026-09-27 10:55:49 on dist-test-slave-tz51"
server_key: "89fd3b506f389ee6418229b39e8b2f1d"
server_key_iv: "53f3fa6bf93b36b44a36027cbf2e9308"
server_key_version: "encryptionkey@0"
I20260927 10:55:49.226580 28078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-2/wal/instance:
uuid: "bcda15a0034242a1b3039a562bbf84d7"
format_stamp: "Formatted at 2026-09-27 10:55:49 on dist-test-slave-tz51"
server_key: "89fd3b506f389ee6418229b39e8b2f1d"
server_key_iv: "53f3fa6bf93b36b44a36027cbf2e9308"
server_key_version: "encryptionkey@0"
I20260927 10:55:49.230194 28078 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:55:49.232686 28094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:49.233851 28078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:55:49.233987 28078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-2/wal
uuid: "bcda15a0034242a1b3039a562bbf84d7"
format_stamp: "Formatted at 2026-09-27 10:55:49 on dist-test-slave-tz51"
server_key: "89fd3b506f389ee6418229b39e8b2f1d"
server_key_iv: "53f3fa6bf93b36b44a36027cbf2e9308"
server_key_version: "encryptionkey@0"
I20260927 10:55:49.234088 28078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:49.259919 28078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:49.263290 28078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:49.263499 28078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:49.270464 28078 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.60:46131
I20260927 10:55:49.270485 28146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.60:46131 every 8 connection(s)
I20260927 10:55:49.271409 28078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-2/data/info.pb
I20260927 10:55:49.274173 28147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:49.277822 28147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45875 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:55:49.280278 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 28078
I20260927 10:55:49.280438 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506389460539-14608-0/minicluster-data/master-2/wal/instance
I20260927 10:55:49.280745 14608 external_mini_cluster.cc:1428] Setting key a3d7117a4512b4cc6ba80399b4a10537
Sep 27 10:55:49 dist-test-slave-tz51 krb5kdc[27915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506549, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.60@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
Sep 27 10:55:49 dist-test-slave-tz51 krb5kdc[27915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.60@KRBTEST.COM
I20260927 10:55:49.291280 27998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.279691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56740 (local address 127.14.68.62:45875)
0927 10:55:49.279882 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:49.279886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:49.280813 (+ 927us) server_negotiation.cc:407] Connection header received
0927 10:55:49.281926 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:49.281930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:49.281989 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:49.282091 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:49.283010 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.283520 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.284557 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.284784 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.284896 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:49.287515 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:49.287519 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:49.287522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:49.287553 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:49.289101 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.289608 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.289611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.289612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.289693 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.290053 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.290057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.290059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.290317 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:49.290390 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:49.290977 (+ 587us) server_negotiation.cc:299] Negotiation successful
0927 10:55:49.291096 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:55:49.292260 28151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.281448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49660 (local address 127.14.68.60:46131)
0927 10:55:49.281803 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:49.281815 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:49.281867 (+ 52us) server_negotiation.cc:407] Connection header received
0927 10:55:49.282004 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:49.282008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:49.282173 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:49.282348 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:49.283137 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.284014 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.284813 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.285244 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.285363 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:49.287951 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:49.287959 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:49.287974 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:49.288003 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:49.289832 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.290287 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.290291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.290292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.290382 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.290633 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.290637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.290638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.290884 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:49.291051 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:49.291287 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 10:55:49.291402 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":261,"spinlock_wait_cycles":2944,"thread_start_us":144,"threads_started":1}
I20260927 10:55:49.292260 28149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.280091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:45875 (local address 127.0.0.1:56740)
0927 10:55:49.280633 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:49.280692 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:49.281707 (+ 1015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:49.282173 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:49.282184 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:49.282526 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:49.282873 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:49.282884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.283652 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.283656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:49.284435 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:49.284444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.284930 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.284935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:49.285134 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:49.285750 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:49.285775 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:49.287376 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:49.289206 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:49.289212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:49.289221 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:49.289511 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:49.289781 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:49.289787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:49.289792 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:49.289955 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:49.290410 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:49.290419 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:49.290708 (+ 289us) client_negotiation.cc:769] Sending connection context
0927 10:55:49.290905 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 10:55:49.291250 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":5,"thread_start_us":122,"threads_started":1}
I20260927 10:55:49.293550 28147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:55:49 dist-test-slave-tz51 krb5kdc[27915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506549, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.60@KRBTEST.COM for kudu/127.14.68.61@KRBTEST.COM
Sep 27 10:55:49 dist-test-slave-tz51 krb5kdc[27915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:55:49.302073 28149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.294125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:37097 (local address 127.0.0.1:44656)
0927 10:55:49.294245 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:49.294260 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:49.294329 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:49.294690 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:49.294692 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:49.294813 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:49.294891 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:49.294895 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.295615 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.295621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:49.296213 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:49.296221 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.296695 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.296699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:49.296792 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:49.297266 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:49.297284 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:49.298500 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:49.300653 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:49.300655 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:49.300657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:49.300831 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:49.301115 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:49.301118 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:49.301120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:49.301234 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:49.301647 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:49.301650 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:49.301716 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 10:55:49.301810 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 10:55:49.301929 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 10:55:49.302172 28077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.294263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44656 (local address 127.14.68.61:37097)
0927 10:55:49.294400 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:49.294405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:49.294418 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:55:49.294459 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:49.294462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:49.294506 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:49.294581 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:49.295009 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.295509 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.296336 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.296583 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.296699 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:49.298638 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:49.298640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:49.298642 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:49.298662 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:49.300539 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.300954 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.300957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.300958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.301027 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.301324 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.301327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.301329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.301549 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:49.301658 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:49.301883 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 10:55:49.302004 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 10:55:49.302623 27998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.294505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56744 (local address 127.14.68.62:45875)
0927 10:55:49.294629 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:49.294633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:49.294648 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:55:49.294709 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:49.294712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:49.294747 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:49.294835 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:49.295373 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.295903 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.296650 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.296872 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.296990 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:49.299205 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:49.299211 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:49.299216 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:49.299246 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:49.300814 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.301232 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.301234 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.301235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.301326 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.301580 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.301586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.301591 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.301830 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:49.302033 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:49.302390 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 10:55:49.302511 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:55:49.302973 28147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:55:49.306147 27996 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } attempt: 3
Sep 27 10:55:49 dist-test-slave-tz51 krb5kdc[27915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506549, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.60@KRBTEST.COM for kudu/127.14.68.60@KRBTEST.COM
I20260927 10:55:49.308900 28071 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } attempt: 3
Sep 27 10:55:49 dist-test-slave-tz51 krb5kdc[27915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.61@KRBTEST.COM
I20260927 10:55:49.311085 28149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.303439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:46131 (local address 127.0.0.1:49668)
0927 10:55:49.303529 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:49.303543 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:49.303624 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:49.303851 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:49.303853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:49.304007 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:49.304062 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:49.304067 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.304764 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.304767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:49.305254 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:49.305261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.305713 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.305716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:49.305789 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:49.306219 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:49.306233 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:49.307629 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:49.309580 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:49.309583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:49.309585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:49.309876 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:49.310193 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:49.310197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:49.310200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:49.310287 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:49.310633 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:49.310636 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:49.310739 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 10:55:49.310845 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 10:55:49.310974 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":3}
I20260927 10:55:49.311543 28151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.303496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49668 (local address 127.14.68.60:46131)
0927 10:55:49.303596 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:49.303599 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:49.303611 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:55:49.303717 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:49.303721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:49.303754 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:49.303841 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:49.304165 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.304658 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.305368 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.305611 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.305713 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:49.307776 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:49.307780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:49.307782 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:49.307803 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:49.309474 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.309999 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.310003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.310008 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.310088 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.310374 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.310377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.310379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.310532 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:49.310641 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:49.311353 (+ 712us) server_negotiation.cc:299] Negotiation successful
0927 10:55:49.311452 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 10:55:49 dist-test-slave-tz51 krb5kdc[27915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506548, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for kudu/127.14.68.60@KRBTEST.COM
I20260927 10:55:49.314374 28077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.303681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44672 (local address 127.14.68.61:37097)
0927 10:55:49.303800 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:49.303804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:49.303884 (+ 80us) server_negotiation.cc:407] Connection header received
0927 10:55:49.303990 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:49.303994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:49.304029 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:49.304120 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:49.304811 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.305281 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.306043 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.306605 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.306719 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:49.309840 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:49.309844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:49.309847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:49.309869 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:49.312703 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.313137 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.313140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.313142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.313223 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.313495 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.313498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.313500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.313711 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:49.313903 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:49.314093 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 10:55:49.314215 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:55:49.315316 28147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcda15a0034242a1b3039a562bbf84d7: Bootstrap starting.
Sep 27 10:55:49 dist-test-slave-tz51 krb5kdc[27915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506549, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.61@KRBTEST.COM for kudu/127.14.68.60@KRBTEST.COM
I20260927 10:55:49.318069 28147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcda15a0034242a1b3039a562bbf84d7: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:49.318496 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:55:49.318960 28147 log.cc:824] T 00000000000000000000000000000000 P bcda15a0034242a1b3039a562bbf84d7: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:49.319522 28074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.309286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:46131 (local address 127.0.0.1:49682)
0927 10:55:49.309515 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:49.309531 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:49.309596 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:49.311377 (+ 1781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:49.311382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:49.311565 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:49.311658 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:49.311665 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.312830 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.312834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:49.313548 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:49.313559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.314160 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.314163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:49.314245 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:49.314668 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:49.314682 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:49.315964 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:49.317951 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:49.317954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:49.317956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:49.318182 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:49.318475 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:49.318477 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:49.318479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:49.318594 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:49.318996 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:49.319000 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:49.319088 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 10:55:49.319244 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 10:55:49.319371 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260927 10:55:49.319811 28153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.309771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49682 (local address 127.14.68.60:46131)
0927 10:55:49.311121 (+ 1350us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:49.311144 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:49.311157 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:55:49.311194 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:49.311198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:49.311250 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:49.311786 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:49.311923 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.312720 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.313679 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.314034 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.314568 (+ 534us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:49.316125 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:49.316128 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:49.316130 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:49.316153 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:49.317759 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.318279 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.318284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.318286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.318369 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.318709 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.318711 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.318712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.318879 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:49.318982 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:49.319305 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 10:55:49.319406 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1273,"thread_start_us":120,"threads_started":1}
I20260927 10:55:49.321822 28147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcda15a0034242a1b3039a562bbf84d7: No bootstrap required, opened a new log
I20260927 10:55:49.322726 28071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12: Bootstrap starting.
I20260927 10:55:49.325196 28071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:49.326086 28071 log.cc:824] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:49.326767 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.306527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:46131 (local address 127.0.0.1:49680)
0927 10:55:49.306711 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:49.306725 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:49.306794 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:49.307159 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:49.307163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:49.307336 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:49.307437 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:49.307445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.309359 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.309362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:49.310182 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:49.310194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.310871 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.310875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:49.310959 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:49.311429 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:49.311444 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:49.319664 (+ 8220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:49.325222 (+ 5558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:49.325228 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:49.325233 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:49.325479 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:49.325781 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:49.325784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:49.325786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:49.325866 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:49.326300 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:49.326303 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:49.326390 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 10:55:49.326485 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 10:55:49.326618 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 10:55:49.327221 28152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.306564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.14.68.60:46131)
0927 10:55:49.306956 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:49.306962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:49.306971 (+ 9us) server_negotiation.cc:407] Connection header received
0927 10:55:49.306995 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:49.306998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:49.307026 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:49.307128 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:49.307561 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.309222 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.310346 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.310746 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.310869 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:49.319864 (+ 8995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:49.319870 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:49.319875 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:49.319904 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:49.325090 (+ 5186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.325588 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.325591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.325593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.325686 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.325953 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.325956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.325957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.326179 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:49.326259 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:49.326942 (+ 683us) server_negotiation.cc:299] Negotiation successful
0927 10:55:49.327053 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":114,"threads_started":1}
I20260927 10:55:49.328202 28071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12: No bootstrap required, opened a new log
I20260927 10:55:49.330237 28147 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcda15a0034242a1b3039a562bbf84d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e47af1c5134896b489f95003d4cafd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45875 } } peers { permanent_uuid: "4c7e2fb24a894b158870888bd4ee5a12" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } } peers { permanent_uuid: "bcda15a0034242a1b3039a562bbf84d7" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } }
I20260927 10:55:49.330509 28147 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcda15a0034242a1b3039a562bbf84d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:49.332420 28071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e47af1c5134896b489f95003d4cafd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45875 } } peers { permanent_uuid: "4c7e2fb24a894b158870888bd4ee5a12" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } } peers { permanent_uuid: "bcda15a0034242a1b3039a562bbf84d7" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } }
I20260927 10:55:49.332680 28071 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:49.332738 28071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c7e2fb24a894b158870888bd4ee5a12, State: Initialized, Role: FOLLOWER
I20260927 10:55:49.333042 27996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e47af1c5134896b489f95003d4cafd: Bootstrap starting.
I20260927 10:55:49.333279 27998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.323963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56754 (local address 127.14.68.62:45875)
0927 10:55:49.324092 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:49.324113 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:49.324129 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:55:49.324165 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:49.324169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:49.324217 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:49.324302 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:49.325299 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.326019 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.327153 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.327463 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.327561 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:49.329203 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:49.329206 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:49.329209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:49.329238 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:49.331024 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.331491 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.331494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.331496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.331572 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.331915 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.331919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.331920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.332369 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:49.332485 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:49.333046 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 10:55:49.333155 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 10:55:49.333195 28071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e47af1c5134896b489f95003d4cafd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45875 } } peers { permanent_uuid: "4c7e2fb24a894b158870888bd4ee5a12" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } } peers { permanent_uuid: "bcda15a0034242a1b3039a562bbf84d7" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } }
I20260927 10:55:49.333706 28163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e47af1c5134896b489f95003d4cafd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45875 } } peers { permanent_uuid: "4c7e2fb24a894b158870888bd4ee5a12" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } } peers { permanent_uuid: "bcda15a0034242a1b3039a562bbf84d7" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } } }
I20260927 10:55:49.333829 28163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:55:49.334601 28071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:55:49.335484 27996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52e47af1c5134896b489f95003d4cafd: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:49.339902 28153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.321035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49698 (local address 127.14.68.60:46131)
0927 10:55:49.321139 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:49.321143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:49.324303 (+ 3160us) server_negotiation.cc:407] Connection header received
0927 10:55:49.325373 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:49.325378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:49.325430 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:49.325484 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:49.326815 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.327565 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.331214 (+ 3649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.331563 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.331661 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:49.335652 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:49.335658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:49.335662 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:49.335691 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:49.338012 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.338488 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.338491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.338493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.338569 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.338865 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.338868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.338870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.339102 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:49.339229 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:49.339593 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 10:55:49.339753 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 10:55:49.340104 28077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:49.323855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44682 (local address 127.14.68.61:37097)
0927 10:55:49.324075 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:49.324080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:49.324487 (+ 407us) server_negotiation.cc:407] Connection header received
0927 10:55:49.324733 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:49.324736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:49.324774 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:49.324819 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:49.329240 (+ 4421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.329937 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.332271 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:49.332579 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:49.332657 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:49.336607 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:49.336614 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:49.336618 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:49.336648 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:49.338288 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.338819 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.338822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.338824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.338900 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:49.339213 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:49.339216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:49.339218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:49.339403 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:49.339501 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:49.339868 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 10:55:49.339976 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
W20260927 10:55:49.340559 28174 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:55:49.340639 28174 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:55:49.341405 27996 log.cc:824] T 00000000000000000000000000000000 P 52e47af1c5134896b489f95003d4cafd: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:49.330569 28147 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcda15a0034242a1b3039a562bbf84d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcda15a0034242a1b3039a562bbf84d7, State: Initialized, Role: FOLLOWER
I20260927 10:55:49.342224 28147 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcda15a0034242a1b3039a562bbf84d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e47af1c5134896b489f95003d4cafd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45875 } } peers { permanent_uuid: "4c7e2fb24a894b158870888bd4ee5a12" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } } peers { permanent_uuid: "bcda15a0034242a1b3039a562bbf84d7" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } }
I20260927 10:55:49.342702 28177 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcda15a0034242a1b3039a562bbf84d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e47af1c5134896b489f95003d4cafd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45875 } } peers { permanent_uuid: "4c7e2fb24a894b158870888bd4ee5a12" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } } peers { permanent_uuid: "bcda15a0034242a1b3039a562bbf84d7" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } } }
I20260927 10:55:49.342830 28177 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcda15a0034242a1b3039a562bbf84d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:55:49.343477 28147 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcda15a0034242a1b3039a562bbf84d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:55:49.341231 28003 master_runner.cc:186] Error getting master registration for 127.14.68.62:45875: OK, instance_id { permanent_uuid: "52e47af1c5134896b489f95003d4cafd" instance_seqno: 1790506548842511 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:55:49.344550 28003 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 10:55:49.349668 28188 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bcda15a0034242a1b3039a562bbf84d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 10:55:49.346705 28078 master_runner.cc:186] Error getting master registration for 127.14.68.62:45875: OK, instance_id { permanent_uuid: "52e47af1c5134896b489f95003d4cafd" instance_seqno: 1790506548842511 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260927 10:55:49.349753 28188 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:55:49.349848 28078 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:55:49.351013 27996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e47af1c5134896b489f95003d4cafd: No bootstrap required, opened a new log
I20260927 10:55:49.355070 27996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52e47af1c5134896b489f95003d4cafd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e47af1c5134896b489f95003d4cafd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45875 } } peers { permanent_uuid: "4c7e2fb24a894b158870888bd4ee5a12" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } } peers { permanent_uuid: "bcda15a0034242a1b3039a562bbf84d7" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } }
I20260927 10:55:49.355319 27996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52e47af1c5134896b489f95003d4cafd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:49.355381 27996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52e47af1c5134896b489f95003d4cafd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52e47af1c5134896b489f95003d4cafd, State: Initialized, Role: FOLLOWER
I20260927 10:55:49.355839 27996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52e47af1c5134896b489f95003d4cafd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e47af1c5134896b489f95003d4cafd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45875 } } peers { permanent_uuid: "4c7e2fb24a894b158870888bd4ee5a12" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } } peers { permanent_uuid: "bcda15a0034242a1b3039a562bbf84d7" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } }
I20260927 10:55:49.356359 28190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e47af1c5134896b489f95003d4cafd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e47af1c5134896b489f95003d4cafd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45875 } } peers { permanent_uuid: "4c7e2fb24a894b158870888bd4ee5a12" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } } peers { permanent_uuid: "bcda15a0034242a1b3039a562bbf84d7" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } } }
I20260927 10:55:49.356489 28190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e47af1c5134896b489f95003d4cafd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:55:49.357174 27996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52e47af1c5134896b489f95003d4cafd [sys.catalog]: configured and running, proceeding with master startup.
W20260927 10:55:49.361888 28201 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 52e47af1c5134896b489f95003d4cafd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:55:49.361965 28201 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:55:49.364691 28163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:55:49.364831 28163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e47af1c5134896b489f95003d4cafd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45875 } } peers { permanent_uuid: "4c7e2fb24a894b158870888bd4ee5a12" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } } peers { permanent_uuid: "bcda15a0034242a1b3039a562bbf84d7" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } }
I20260927 10:55:49.365702 28163 leader_election.cc:288] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52e47af1c5134896b489f95003d4cafd (127.14.68.62:45875), bcda15a0034242a1b3039a562bbf84d7 (127.14.68.60:46131)
I20260927 10:55:49.366329 27970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c7e2fb24a894b158870888bd4ee5a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52e47af1c5134896b489f95003d4cafd" is_pre_election: true
I20260927 10:55:49.366588 27970 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 52e47af1c5134896b489f95003d4cafd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c7e2fb24a894b158870888bd4ee5a12 in term 0.
I20260927 10:55:49.366976 28019 leader_election.cc:302] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c7e2fb24a894b158870888bd4ee5a12, 52e47af1c5134896b489f95003d4cafd; no voters:
I20260927 10:55:49.367542 28121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c7e2fb24a894b158870888bd4ee5a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcda15a0034242a1b3039a562bbf84d7" is_pre_election: true
I20260927 10:55:49.367832 28121 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bcda15a0034242a1b3039a562bbf84d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c7e2fb24a894b158870888bd4ee5a12 in term 0.
I20260927 10:55:49.369961 27928 master_runner.cc:263] No leader master found from master 52e47af1c5134896b489f95003d4cafd
I20260927 10:55:49.370028 27928 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:55:49.370600 28163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:55:49.370692 28163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:55:49.370750 28163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:49.371990 28163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e47af1c5134896b489f95003d4cafd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45875 } } peers { permanent_uuid: "4c7e2fb24a894b158870888bd4ee5a12" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } } peers { permanent_uuid: "bcda15a0034242a1b3039a562bbf84d7" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } }
I20260927 10:55:49.372344 28163 leader_election.cc:288] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [CANDIDATE]: Term 1 election: Requested vote from peers 52e47af1c5134896b489f95003d4cafd (127.14.68.62:45875), bcda15a0034242a1b3039a562bbf84d7 (127.14.68.60:46131)
I20260927 10:55:49.372874 27970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c7e2fb24a894b158870888bd4ee5a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52e47af1c5134896b489f95003d4cafd"
I20260927 10:55:49.372994 27970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52e47af1c5134896b489f95003d4cafd [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:49.374200 27970 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 52e47af1c5134896b489f95003d4cafd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c7e2fb24a894b158870888bd4ee5a12 in term 1.
I20260927 10:55:49.374488 28019 leader_election.cc:302] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c7e2fb24a894b158870888bd4ee5a12, 52e47af1c5134896b489f95003d4cafd; no voters:
I20260927 10:55:49.374688 28121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c7e2fb24a894b158870888bd4ee5a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcda15a0034242a1b3039a562bbf84d7"
I20260927 10:55:49.374806 28121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcda15a0034242a1b3039a562bbf84d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:49.376283 28163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:55:49.376495 28163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [term 1 LEADER]: Becoming Leader. State: Replica: 4c7e2fb24a894b158870888bd4ee5a12, State: Running, Role: LEADER
I20260927 10:55:49.378273 28121 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bcda15a0034242a1b3039a562bbf84d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c7e2fb24a894b158870888bd4ee5a12 in term 1.
I20260927 10:55:49.379199 28163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e47af1c5134896b489f95003d4cafd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45875 } } peers { permanent_uuid: "4c7e2fb24a894b158870888bd4ee5a12" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } } peers { permanent_uuid: "bcda15a0034242a1b3039a562bbf84d7" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } }
I20260927 10:55:49.381764 28163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c7e2fb24a894b158870888bd4ee5a12. Latest consensus state: current_term: 1 leader_uuid: "4c7e2fb24a894b158870888bd4ee5a12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e47af1c5134896b489f95003d4cafd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45875 } } peers { permanent_uuid: "4c7e2fb24a894b158870888bd4ee5a12" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } } peers { permanent_uuid: "bcda15a0034242a1b3039a562bbf84d7" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } } }
I20260927 10:55:49.381875 28163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:49.391956 28205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:55:49.392910 28205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:55:49.400108 28121 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bcda15a0034242a1b3039a562bbf84d7 [term 1 FOLLOWER]: Refusing update from remote peer 4c7e2fb24a894b158870888bd4ee5a12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:55:49.402601 28203 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcda15a0034242a1b3039a562bbf84d7" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:49.407048 28177 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcda15a0034242a1b3039a562bbf84d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c7e2fb24a894b158870888bd4ee5a12. Latest consensus state: current_term: 1 leader_uuid: "4c7e2fb24a894b158870888bd4ee5a12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e47af1c5134896b489f95003d4cafd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45875 } } peers { permanent_uuid: "4c7e2fb24a894b158870888bd4ee5a12" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } } peers { permanent_uuid: "bcda15a0034242a1b3039a562bbf84d7" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } } }
I20260927 10:55:49.407187 28177 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcda15a0034242a1b3039a562bbf84d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:55:49.408123 27970 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 52e47af1c5134896b489f95003d4cafd [term 1 FOLLOWER]: Refusing update from remote peer 4c7e2fb24a894b158870888bd4ee5a12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:55:49.408669 28203 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52e47af1c5134896b489f95003d4cafd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:49.412673 28190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e47af1c5134896b489f95003d4cafd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c7e2fb24a894b158870888bd4ee5a12. Latest consensus state: current_term: 1 leader_uuid: "4c7e2fb24a894b158870888bd4ee5a12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e47af1c5134896b489f95003d4cafd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45875 } } peers { permanent_uuid: "4c7e2fb24a894b158870888bd4ee5a12" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } } peers { permanent_uuid: "bcda15a0034242a1b3039a562bbf84d7" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } } }
I20260927 10:55:49.412792 28190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e47af1c5134896b489f95003d4cafd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:55:49.417114 28203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c7e2fb24a894b158870888bd4ee5a12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e47af1c5134896b489f95003d4cafd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45875 } } peers { permanent_uuid: "4c7e2fb24a894b158870888bd4ee5a12" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } } peers { permanent_uuid: "bcda15a0034242a1b3039a562bbf84d7" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } } }
I20260927 10:55:49.417243 28203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:49.418763 28190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e47af1c5134896b489f95003d4cafd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c7e2fb24a894b158870888bd4ee5a12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e47af1c5134896b489f95003d4cafd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45875 } } peers { permanent_uuid: "4c7e2fb24a894b158870888bd4ee5a12" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } } peers { permanent_uuid: "bcda15a0034242a1b3039a562bbf84d7" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } } }
I20260927 10:55:49.418874 28190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e47af1c5134896b489f95003d4cafd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:55:49.419301 28163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c7e2fb24a894b158870888bd4ee5a12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e47af1c5134896b489f95003d4cafd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45875 } } peers { permanent_uuid: "4c7e2fb24a894b158870888bd4ee5a12" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } } peers { permanent_uuid: "bcda15a0034242a1b3039a562bbf84d7" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } } }
I20260927 10:55:49.419406 28163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:49.420680 28177 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcda15a0034242a1b3039a562bbf84d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c7e2fb24a894b158870888bd4ee5a12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e47af1c5134896b489f95003d4cafd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 45875 } } peers { permanent_uuid: "4c7e2fb24a894b158870888bd4ee5a12" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 37097 } } peers { permanent_uuid: "bcda15a0034242a1b3039a562bbf84d7" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 46131 } } }
I20260927 10:55:49.420794 28177 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcda15a0034242a1b3039a562bbf84d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:55:49.421207 28207 mvcc.cc:205] Tried to move back new op lower bound from 7333914826335023104 to 7333914826260156416. Current Snapshot: MvccSnapshot[applied={T|T < 7333914826335023104}]
I20260927 10:55:49.421376 28209 mvcc.cc:205] Tried to move back new op lower bound from 7333914826335023104 to 7333914826260156416. Current Snapshot: MvccSnapshot[applied={T|T < 7333914826335023104}]
I20260927 10:55:49.421521 28206 mvcc.cc:205] Tried to move back new op lower bound from 7333914826335023104 to 7333914826260156416. Current Snapshot: MvccSnapshot[applied={T|T < 7333914826335023104}]
I20260927 10:55:49.425042 28205 catalog_manager.cc:1347] Generated new cluster ID: cf1a512f13674d01a37f558db5fadee6
I20260927 10:55:49.425107 28205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:55:49.465095 28205 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:55:49.466272 28205 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:55:49.476523 28205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c7e2fb24a894b158870888bd4ee5a12: Generated new TSK 0
I20260927 10:55:49.477324 28205 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:55:50.351028 28188 catalog_manager.cc:1259] Loaded cluster ID: cf1a512f13674d01a37f558db5fadee6
I20260927 10:55:50.351105 28188 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bcda15a0034242a1b3039a562bbf84d7: loading cluster ID for follower catalog manager: success
I20260927 10:55:50.352779 28188 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bcda15a0034242a1b3039a562bbf84d7: acquiring CA information for follower catalog manager: success
I20260927 10:55:50.353875 28188 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bcda15a0034242a1b3039a562bbf84d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:55:50.363019 28201 catalog_manager.cc:1259] Loaded cluster ID: cf1a512f13674d01a37f558db5fadee6
I20260927 10:55:50.363076 28201 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 52e47af1c5134896b489f95003d4cafd: loading cluster ID for follower catalog manager: success
I20260927 10:55:50.364563 28201 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 52e47af1c5134896b489f95003d4cafd: acquiring CA information for follower catalog manager: success
I20260927 10:55:50.365618 28201 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 52e47af1c5134896b489f95003d4cafd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:55:50.539495 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 27928
I20260927 10:55:50.545933 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 28003
I20260927 10:55:50.551462 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 28078
2026-09-27T10:55:50Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3040 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:55:50 dist-test-slave-tz51 krb5kdc[28221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:55:50 dist-test-slave-tz51 krb5kdc[28221](info): set up 2 sockets
Sep 27 10:55:50 dist-test-slave-tz51 krb5kdc[28221](info): commencing operation
krb5kdc: starting...
W20260927 10:55:51.592072 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:55:51 dist-test-slave-tz51 krb5kdc[28221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:55:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:55:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:55:51.711192 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:35859
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:41725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:35859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:51.817229 28234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:51.817476 28234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:51.817512 28234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:51.821342 28234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:55:51.821389 28234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:51.821410 28234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:51.821426 28234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:55:51.821441 28234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:55:51.826897 28234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:41725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:35859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:35859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:51.827868 28234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:51.828933 28234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:51.836906 28240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:51.836920 28239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:51.836920 28242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:51.837133 28234 server_base.cc:1034] running on GCE node
I20260927 10:55:51.837704 28234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:51.838745 28234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:51.839901 28234 hybrid_clock.cc:648] HybridClock initialized: now 1790506551839886 us; error 28 us; skew 500 ppm
Sep 27 10:55:51 dist-test-slave-tz51 krb5kdc[28221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506551, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:51.842464 28234 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:55:51.843225 28234 webserver.cc:460] Webserver started at http://127.14.68.62:38877/ using document root <none> and password file <none>
I20260927 10:55:51.843776 28234 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:51.843842 28234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:51.844000 28234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:51.845897 28234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "27ef682c3fb44383a901c2a9ad8a0546"
format_stamp: "Formatted at 2026-09-27 10:55:51 on dist-test-slave-tz51"
server_key: "468aadb6cd9c096da36a6d88f87e51f5"
server_key_iv: "ed6e4b4732b9311f3c5efb147e916413"
server_key_version: "encryptionkey@0"
I20260927 10:55:51.846423 28234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "27ef682c3fb44383a901c2a9ad8a0546"
format_stamp: "Formatted at 2026-09-27 10:55:51 on dist-test-slave-tz51"
server_key: "468aadb6cd9c096da36a6d88f87e51f5"
server_key_iv: "ed6e4b4732b9311f3c5efb147e916413"
server_key_version: "encryptionkey@0"
I20260927 10:55:51.849865 28234 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:55:51.852073 28249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:51.853212 28234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:55:51.853349 28234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "27ef682c3fb44383a901c2a9ad8a0546"
format_stamp: "Formatted at 2026-09-27 10:55:51 on dist-test-slave-tz51"
server_key: "468aadb6cd9c096da36a6d88f87e51f5"
server_key_iv: "ed6e4b4732b9311f3c5efb147e916413"
server_key_version: "encryptionkey@0"
I20260927 10:55:51.853458 28234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:51.873065 28234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:51.875728 28234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:51.875921 28234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:51.882495 28234 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:35859
I20260927 10:55:51.882506 28301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:35859 every 8 connection(s)
I20260927 10:55:51.883373 28234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:55:51.885983 28302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:51.886837 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 28234
I20260927 10:55:51.886951 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:55:51.887274 14608 external_mini_cluster.cc:1428] Setting key 6ca0879ce7b62347894047a2d2547bdf
I20260927 10:55:51.891963 28302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27ef682c3fb44383a901c2a9ad8a0546: Bootstrap starting.
I20260927 10:55:51.894088 28302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27ef682c3fb44383a901c2a9ad8a0546: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:51.894804 28302 log.cc:824] T 00000000000000000000000000000000 P 27ef682c3fb44383a901c2a9ad8a0546: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:55:51 dist-test-slave-tz51 krb5kdc[28221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:55:51.896669 28302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27ef682c3fb44383a901c2a9ad8a0546: No bootstrap required, opened a new log
I20260927 10:55:51.900068 28302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27ef682c3fb44383a901c2a9ad8a0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ef682c3fb44383a901c2a9ad8a0546" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 35859 } }
I20260927 10:55:51.900305 28302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27ef682c3fb44383a901c2a9ad8a0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:51.900384 28302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27ef682c3fb44383a901c2a9ad8a0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ef682c3fb44383a901c2a9ad8a0546, State: Initialized, Role: FOLLOWER
I20260927 10:55:51.900693 28305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:51.888594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60860 (local address 127.14.68.62:35859)
0927 10:55:51.889017 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:51.889031 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:51.889065 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:55:51.889698 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:51.889714 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:51.890015 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:51.890347 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:51.891145 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:51.892226 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:51.893080 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:51.893535 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:51.893713 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:51.896186 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:51.896190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:51.896203 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:51.896227 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:51.897864 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:51.898307 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:51.898318 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:51.898322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:51.898426 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:51.898710 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:51.898714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:51.898715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:51.899067 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:51.899276 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:51.899588 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 10:55:51.899886 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":124,"threads_started":1}
I20260927 10:55:51.900892 28302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27ef682c3fb44383a901c2a9ad8a0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ef682c3fb44383a901c2a9ad8a0546" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 35859 } }
I20260927 10:55:51.901046 28302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27ef682c3fb44383a901c2a9ad8a0546 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:55:51.901110 28302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27ef682c3fb44383a901c2a9ad8a0546 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:55:51.901211 28302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27ef682c3fb44383a901c2a9ad8a0546 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:51.902117 28302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27ef682c3fb44383a901c2a9ad8a0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ef682c3fb44383a901c2a9ad8a0546" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 35859 } }
I20260927 10:55:51.902490 28302 leader_election.cc:302] T 00000000000000000000000000000000 P 27ef682c3fb44383a901c2a9ad8a0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27ef682c3fb44383a901c2a9ad8a0546; no voters:
I20260927 10:55:51.902796 28302 leader_election.cc:288] T 00000000000000000000000000000000 P 27ef682c3fb44383a901c2a9ad8a0546 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:55:51.902909 28307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27ef682c3fb44383a901c2a9ad8a0546 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:55:51.903117 28307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27ef682c3fb44383a901c2a9ad8a0546 [term 1 LEADER]: Becoming Leader. State: Replica: 27ef682c3fb44383a901c2a9ad8a0546, State: Running, Role: LEADER
I20260927 10:55:51.903558 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:55:51.904034 28302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27ef682c3fb44383a901c2a9ad8a0546 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:55:51.903460 28307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27ef682c3fb44383a901c2a9ad8a0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ef682c3fb44383a901c2a9ad8a0546" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 35859 } }
I20260927 10:55:51.906474 28308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27ef682c3fb44383a901c2a9ad8a0546 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27ef682c3fb44383a901c2a9ad8a0546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ef682c3fb44383a901c2a9ad8a0546" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 35859 } } }
I20260927 10:55:51.906632 28308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27ef682c3fb44383a901c2a9ad8a0546 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:51.906598 28309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27ef682c3fb44383a901c2a9ad8a0546 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27ef682c3fb44383a901c2a9ad8a0546. Latest consensus state: current_term: 1 leader_uuid: "27ef682c3fb44383a901c2a9ad8a0546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ef682c3fb44383a901c2a9ad8a0546" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 35859 } } }
I20260927 10:55:51.906678 28309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27ef682c3fb44383a901c2a9ad8a0546 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:51.907022 28322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:55:51.910804 28322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:55:51.916666 28322 catalog_manager.cc:1347] Generated new cluster ID: a061f7d2165540a3ad7008f223e23b14
I20260927 10:55:51.916739 28322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:55:51.925727 28322 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:55:51.926643 28322 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:55:51.933440 28322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27ef682c3fb44383a901c2a9ad8a0546: Generated new TSK 0
I20260927 10:55:51.934055 28322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:55:51.962445 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 28234
2026-09-27T10:55:51Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1409 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:55:52 dist-test-slave-tz51 krb5kdc[28329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:55:52 dist-test-slave-tz51 krb5kdc[28329](info): set up 2 sockets
Sep 27 10:55:52 dist-test-slave-tz51 krb5kdc[28329](info): commencing operation
krb5kdc: starting...
W20260927 10:55:52.991961 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:55:53 dist-test-slave-tz51 krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:55:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:55:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:55:53.110841 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:40033
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:40033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:53.216876 28342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:53.217159 28342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:53.217226 28342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:53.221165 28342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:55:53.221249 28342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:53.221288 28342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:53.221346 28342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:55:53.221386 28342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:55:53.226439 28342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:40033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:40033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:53.227411 28342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:53.228400 28342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:53.236668 28347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:53.236855 28342 server_base.cc:1034] running on GCE node
W20260927 10:55:53.236693 28350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:53.237047 28348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:53.237568 28342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:53.238647 28342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:53.239781 28342 hybrid_clock.cc:648] HybridClock initialized: now 1790506553239750 us; error 39 us; skew 500 ppm
Sep 27 10:55:53 dist-test-slave-tz51 krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506553, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:53.242374 28342 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:55:53.243228 28342 webserver.cc:460] Webserver started at http://127.14.68.62:46313/ using document root <none> and password file <none>
I20260927 10:55:53.243796 28342 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:53.243860 28342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:53.244026 28342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:53.245968 28342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "b6f6b89832ff479c89587f0802e98250"
format_stamp: "Formatted at 2026-09-27 10:55:53 on dist-test-slave-tz51"
server_key: "fc34ea30f323a52b6d049a8b5fa995d7"
server_key_iv: "3150a5635f4248f605ebe05d3db2d104"
server_key_version: "encryptionkey@0"
I20260927 10:55:53.246488 28342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "b6f6b89832ff479c89587f0802e98250"
format_stamp: "Formatted at 2026-09-27 10:55:53 on dist-test-slave-tz51"
server_key: "fc34ea30f323a52b6d049a8b5fa995d7"
server_key_iv: "3150a5635f4248f605ebe05d3db2d104"
server_key_version: "encryptionkey@0"
I20260927 10:55:53.250022 28342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:55:53.252229 28357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:53.253388 28342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:55:53.253526 28342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "b6f6b89832ff479c89587f0802e98250"
format_stamp: "Formatted at 2026-09-27 10:55:53 on dist-test-slave-tz51"
server_key: "fc34ea30f323a52b6d049a8b5fa995d7"
server_key_iv: "3150a5635f4248f605ebe05d3db2d104"
server_key_version: "encryptionkey@0"
I20260927 10:55:53.253631 28342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:53.275935 28342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:53.278766 28342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:53.278960 28342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:53.285862 28342 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:40033
I20260927 10:55:53.286170 28409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:40033 every 8 connection(s)
I20260927 10:55:53.286819 28342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:55:53.289423 28410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:53.294556 28410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250: Bootstrap starting.
I20260927 10:55:53.296728 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 28342
I20260927 10:55:53.296868 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:55:53.296944 28410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:53.297235 14608 external_mini_cluster.cc:1428] Setting key d61ec01ad9098f01472eb0a17583bffd
I20260927 10:55:53.297807 28410 log.cc:824] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:53.300036 28410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250: No bootstrap required, opened a new log
I20260927 10:55:53.304217 28410 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f6b89832ff479c89587f0802e98250" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40033 } }
I20260927 10:55:53.304476 28410 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:53.304544 28410 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6f6b89832ff479c89587f0802e98250, State: Initialized, Role: FOLLOWER
I20260927 10:55:53.304975 28410 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f6b89832ff479c89587f0802e98250" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40033 } }
I20260927 10:55:53.305094 28410 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:55:53.305177 28410 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:55:53.305279 28410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:53.306133 28410 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f6b89832ff479c89587f0802e98250" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40033 } }
Sep 27 10:55:53 dist-test-slave-tz51 krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:55:53.306483 28410 leader_election.cc:302] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6f6b89832ff479c89587f0802e98250; no voters:
I20260927 10:55:53.306807 28410 leader_election.cc:288] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:55:53.306867 28415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:55:53.307071 28415 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 1 LEADER]: Becoming Leader. State: Replica: b6f6b89832ff479c89587f0802e98250, State: Running, Role: LEADER
I20260927 10:55:53.307729 28415 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f6b89832ff479c89587f0802e98250" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40033 } }
I20260927 10:55:53.308163 28410 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:55:53.309892 28416 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6f6b89832ff479c89587f0802e98250. Latest consensus state: current_term: 1 leader_uuid: "b6f6b89832ff479c89587f0802e98250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f6b89832ff479c89587f0802e98250" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40033 } } }
I20260927 10:55:53.310016 28416 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:53.310328 28417 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6f6b89832ff479c89587f0802e98250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f6b89832ff479c89587f0802e98250" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40033 } } }
I20260927 10:55:53.310402 28417 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:53.312522 28414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:53.298331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41362 (local address 127.14.68.62:40033)
0927 10:55:53.298720 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:53.298740 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:53.298782 (+ 42us) server_negotiation.cc:407] Connection header received
0927 10:55:53.299577 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:53.299598 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:53.300014 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:53.300347 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:53.300978 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:53.302489 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:53.303352 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:53.304091 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:53.304250 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:53.306959 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:53.306966 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:53.306983 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:53.307012 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:53.308920 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:53.309398 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:53.309404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:53.309411 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:53.309564 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:53.309920 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:53.309923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:53.309924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:53.310448 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:53.310648 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:53.311269 (+ 621us) server_negotiation.cc:299] Negotiation successful
0927 10:55:53.311588 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":100,"threads_started":1}
W20260927 10:55:53.313226 28429 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:55:53.313285 28429 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:55:53.313364 28430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:55:53.314059 28430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:55:53.319294 28430 catalog_manager.cc:1347] Generated new cluster ID: ae10e8997f71481fa546527580d14ff9
I20260927 10:55:53.319356 28430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:55:53.332841 28430 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:55:53.334025 28430 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:55:53.344328 28430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250: Generated new TSK 0
I20260927 10:55:53.345039 28430 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
I20260927 10:55:53.379799 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:40033
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:53.503252 28436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:53.503510 28436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:53.503554 28436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:53.507439 28436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:53.507498 28436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:53.507553 28436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:55:53.512705 28436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:40033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:53.513664 28436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:53.514604 28436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:53.523874 28444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:53.523878 28441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:53.523954 28436 server_base.cc:1034] running on GCE node
W20260927 10:55:53.523878 28442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:53.524575 28436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:53.525143 28436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:53.526310 28436 hybrid_clock.cc:648] HybridClock initialized: now 1790506553526280 us; error 42 us; skew 500 ppm
Sep 27 10:55:53 dist-test-slave-tz51 krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506553, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:53.529278 28436 init.cc:377] Logged in from keytab as kudu/127.14.68.1@KRBTEST.COM (short username kudu)
I20260927 10:55:53.530220 28436 webserver.cc:460] Webserver started at http://127.14.68.1:35745/ using document root <none> and password file <none>
I20260927 10:55:53.530939 28436 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:53.531030 28436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:53.531212 28436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:53.533408 28436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "4f8ef97f3e2a467b983129a63d0a3b87"
format_stamp: "Formatted at 2026-09-27 10:55:53 on dist-test-slave-tz51"
server_key: "3fc191b38edd11e56b4ff999d0ae1bfd"
server_key_iv: "180b0882db05b9aba0535396986da20e"
server_key_version: "encryptionkey@0"
I20260927 10:55:53.533938 28436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "4f8ef97f3e2a467b983129a63d0a3b87"
format_stamp: "Formatted at 2026-09-27 10:55:53 on dist-test-slave-tz51"
server_key: "3fc191b38edd11e56b4ff999d0ae1bfd"
server_key_iv: "180b0882db05b9aba0535396986da20e"
server_key_version: "encryptionkey@0"
I20260927 10:55:53.537789 28436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260927 10:55:53.540254 28451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:53.541534 28436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:55:53.541713 28436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "4f8ef97f3e2a467b983129a63d0a3b87"
format_stamp: "Formatted at 2026-09-27 10:55:53 on dist-test-slave-tz51"
server_key: "3fc191b38edd11e56b4ff999d0ae1bfd"
server_key_iv: "180b0882db05b9aba0535396986da20e"
server_key_version: "encryptionkey@0"
I20260927 10:55:53.541828 28436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:53.575899 28436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:53.578958 28436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:53.579175 28436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:53.579838 28436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:53.580662 28436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:53.580747 28436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:53.580833 28436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:53.580879 28436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:53.589919 28436 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:41709
I20260927 10:55:53.589941 28564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:41709 every 8 connection(s)
I20260927 10:55:53.590790 28436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:55:53.596817 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 28436
I20260927 10:55:53.596961 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:55:53.597281 14608 external_mini_cluster.cc:1428] Setting key 15ebbb99a4f73bcf4165d3b3fa8431d7
Sep 27 10:55:53 dist-test-slave-tz51 krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506553, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:55:53.605615 28414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:53.592625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:50005 (local address 127.14.68.62:40033)
0927 10:55:53.592757 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:53.592761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:53.593582 (+ 821us) server_negotiation.cc:407] Connection header received
0927 10:55:53.594923 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:53.594927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:53.594985 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:53.595071 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:53.596278 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:53.597063 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:53.598736 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:53.599088 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:53.599179 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:53.602115 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:53.602119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:53.602122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:53.602151 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:53.603541 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:53.604086 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:53.604090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:53.604091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:53.604175 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:53.604522 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:53.604525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:53.604527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:53.604704 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:53.604816 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:53.605315 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 10:55:53.605444 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:55:53.606415 28567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:53.593001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.1:50005)
0927 10:55:53.593442 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:53.593482 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:53.594718 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:53.595250 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:53.595257 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:53.595598 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:53.596136 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:53.596150 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:53.597187 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:53.597190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:53.598621 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:53.598631 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:53.599257 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:53.599262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:53.599452 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:53.600057 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:53.600082 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:53.601953 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:53.603659 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:53.603665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:53.603679 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:53.603977 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:53.604274 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:53.604277 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:53.604279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:53.604418 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:53.604814 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:53.604821 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:53.605058 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 10:55:53.605258 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 10:55:53.605573 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.14.68.2@KRBTEST.COM; defaulting to no policy
I20260927 10:55:53.607810 28565 heartbeater.cc:347] Connected to a master server at 127.14.68.62:40033
I20260927 10:55:53.608083 28565 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:53.608660 28565 heartbeater.cc:511] Master 127.14.68.62:40033 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.2@KRBTEST.COM" created.
I20260927 10:55:53.610507 28374 ts_manager.cc:194] Registered new tserver with Master: 4f8ef97f3e2a467b983129a63d0a3b87 (127.14.68.1:41709)
I20260927 10:55:53.612077 28374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:50005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
I20260927 10:55:53.627221 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:40033
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:53.736229 28570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:53.736488 28570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:53.736532 28570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:53.740645 28570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:53.740724 28570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:53.740830 28570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:55:53.746304 28570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:40033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:53.747301 28570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:53.748306 28570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:53.757331 28575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:53.757321 28576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:53.757321 28578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:53.757665 28570 server_base.cc:1034] running on GCE node
I20260927 10:55:53.758065 28570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:53.758647 28570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:53.759771 28570 hybrid_clock.cc:648] HybridClock initialized: now 1790506553759735 us; error 42 us; skew 500 ppm
Sep 27 10:55:53 dist-test-slave-tz51 krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506553, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:53.762455 28570 init.cc:377] Logged in from keytab as kudu/127.14.68.2@KRBTEST.COM (short username kudu)
I20260927 10:55:53.763279 28570 webserver.cc:460] Webserver started at http://127.14.68.2:38983/ using document root <none> and password file <none>
I20260927 10:55:53.763823 28570 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:53.763885 28570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:53.764038 28570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:53.765885 28570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "c6b943ae5b344a78941b00aa6af6089a"
format_stamp: "Formatted at 2026-09-27 10:55:53 on dist-test-slave-tz51"
server_key: "19c5ec5b23544aa567062eadc500fe3f"
server_key_iv: "d2b0ddb3fb0485ff8c8481d1271ea80b"
server_key_version: "encryptionkey@0"
I20260927 10:55:53.766388 28570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "c6b943ae5b344a78941b00aa6af6089a"
format_stamp: "Formatted at 2026-09-27 10:55:53 on dist-test-slave-tz51"
server_key: "19c5ec5b23544aa567062eadc500fe3f"
server_key_iv: "d2b0ddb3fb0485ff8c8481d1271ea80b"
server_key_version: "encryptionkey@0"
I20260927 10:55:53.769800 28570 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 10:55:53.772013 28585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:53.773161 28570 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:55:53.773248 28570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "c6b943ae5b344a78941b00aa6af6089a"
format_stamp: "Formatted at 2026-09-27 10:55:53 on dist-test-slave-tz51"
server_key: "19c5ec5b23544aa567062eadc500fe3f"
server_key_iv: "d2b0ddb3fb0485ff8c8481d1271ea80b"
server_key_version: "encryptionkey@0"
I20260927 10:55:53.773324 28570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:53.792856 28570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:53.795970 28570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:53.796105 28570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:53.796576 28570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:53.797276 28570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:53.797320 28570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:53.797358 28570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:53.797379 28570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:53.805781 28570 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:41787
I20260927 10:55:53.805806 28698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:41787 every 8 connection(s)
I20260927 10:55:53.806586 28570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:55:53.813294 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 28570
I20260927 10:55:53.813428 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:55:53.813687 14608 external_mini_cluster.cc:1428] Setting key 33efc671097e608f4d2c0487ef2ad415
Sep 27 10:55:53 dist-test-slave-tz51 krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506553, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:55:53.820823 28414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:53.808285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:43945 (local address 127.14.68.62:40033)
0927 10:55:53.808408 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:53.808412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:53.809434 (+ 1022us) server_negotiation.cc:407] Connection header received
0927 10:55:53.810916 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:53.810919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:53.810967 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:53.811083 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:53.812263 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:53.812798 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:53.814086 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:53.814406 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:53.814491 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:53.816959 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:53.816964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:53.816968 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:53.817001 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:53.818646 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:53.819119 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:53.819125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:53.819130 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:53.819237 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:53.819551 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:53.819558 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:53.819562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:53.819852 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:53.819957 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:53.820438 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 10:55:53.820586 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:55:53.821404 28701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:53.808672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.2:43945)
0927 10:55:53.809253 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:53.809325 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:53.810724 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:53.811271 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:53.811278 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:53.811618 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:53.812121 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:53.812134 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:53.812928 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:53.812932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:53.813938 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:53.813947 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:53.814567 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:53.814571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:53.814758 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:53.815383 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:53.815399 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:53.816793 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:53.818769 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:53.818774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:53.818784 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:53.819014 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:53.819332 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:53.819334 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:53.819335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:53.819453 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:53.819963 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:53.819968 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:53.820189 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 10:55:53.820335 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 10:55:53.820563 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":131,"threads_started":1}
I20260927 10:55:53.822742 28699 heartbeater.cc:347] Connected to a master server at 127.14.68.62:40033
I20260927 10:55:53.823002 28699 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:53.823597 28699 heartbeater.cc:511] Master 127.14.68.62:40033 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.68.3@KRBTEST.COM; defaulting to no policy
I20260927 10:55:53.824775 28374 ts_manager.cc:194] Registered new tserver with Master: c6b943ae5b344a78941b00aa6af6089a (127.14.68.2:41787)
I20260927 10:55:53.825337 28374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:43945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
I20260927 10:55:53.842967 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:40033
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:53.950739 28704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:53.950963 28704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:53.951001 28704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:53.955188 28704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:53.955248 28704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:53.955302 28704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:55:53.960391 28704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:40033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:53.961236 28704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:53.962109 28704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:53.971066 28710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:53.971072 28712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:53.971079 28709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:53.971297 28704 server_base.cc:1034] running on GCE node
I20260927 10:55:53.971822 28704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:53.972404 28704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:53.973618 28704 hybrid_clock.cc:648] HybridClock initialized: now 1790506553973580 us; error 57 us; skew 500 ppm
Sep 27 10:55:53 dist-test-slave-tz51 krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506553, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:53.976493 28704 init.cc:377] Logged in from keytab as kudu/127.14.68.3@KRBTEST.COM (short username kudu)
I20260927 10:55:53.977425 28704 webserver.cc:460] Webserver started at http://127.14.68.3:45513/ using document root <none> and password file <none>
I20260927 10:55:53.977979 28704 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:53.978116 28704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:53.978353 28704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:53.980369 28704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "dfa9d318aaad4575bbf1900757eb8def"
format_stamp: "Formatted at 2026-09-27 10:55:53 on dist-test-slave-tz51"
server_key: "4bad641643f49e925265423519e5564d"
server_key_iv: "8e986a96f39020ed6ada88c97098496f"
server_key_version: "encryptionkey@0"
I20260927 10:55:53.980916 28704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "dfa9d318aaad4575bbf1900757eb8def"
format_stamp: "Formatted at 2026-09-27 10:55:53 on dist-test-slave-tz51"
server_key: "4bad641643f49e925265423519e5564d"
server_key_iv: "8e986a96f39020ed6ada88c97098496f"
server_key_version: "encryptionkey@0"
I20260927 10:55:53.984509 28704 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:55:53.986969 28719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:53.988332 28704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:55:53.988466 28704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "dfa9d318aaad4575bbf1900757eb8def"
format_stamp: "Formatted at 2026-09-27 10:55:53 on dist-test-slave-tz51"
server_key: "4bad641643f49e925265423519e5564d"
server_key_iv: "8e986a96f39020ed6ada88c97098496f"
server_key_version: "encryptionkey@0"
I20260927 10:55:53.988579 28704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:54.001129 28704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:54.004091 28704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:54.004284 28704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:54.004825 28704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:54.005599 28704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:54.005668 28704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:54.005735 28704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:54.005787 28704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:54.014389 28704 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:42187
I20260927 10:55:54.014459 28832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:42187 every 8 connection(s)
I20260927 10:55:54.015300 28704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:55:54.018904 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 28704
I20260927 10:55:54.019078 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:55:54.019361 14608 external_mini_cluster.cc:1428] Setting key 61874e3c69deb4b8784f681f33cf7c67
Sep 27 10:55:54 dist-test-slave-tz51 krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506553, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:55:54.028923 28414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:54.017097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:60503 (local address 127.14.68.62:40033)
0927 10:55:54.017237 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:54.017241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:54.018081 (+ 840us) server_negotiation.cc:407] Connection header received
0927 10:55:54.019579 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:54.019582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:54.019639 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:54.019765 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:54.021298 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.021952 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.023076 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.023291 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.023369 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:54.025691 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:54.025694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:54.025696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:54.025721 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:54.027074 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:54.027533 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:54.027536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:54.027537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:54.027630 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:54.027987 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:54.027989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:54.027990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:54.028171 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:54.028256 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:54.028696 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 10:55:54.028796 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:55:54.029640 28835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:54.017481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.3:60503)
0927 10:55:54.017927 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:54.017968 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:54.019336 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:54.019961 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:54.019968 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:54.020477 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:54.021145 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:54.021164 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.022071 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.022074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:54.022961 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:54.022970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.023416 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.023418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:54.023554 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:54.024084 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:54.024099 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:54.025508 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:54.027187 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:54.027192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:54.027202 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:54.027447 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:54.027759 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:54.027762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:54.027764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:54.027897 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:54.028260 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:54.028265 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:54.028473 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 10:55:54.028623 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 10:55:54.028854 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":122,"threads_started":1}
I20260927 10:55:54.030813 28833 heartbeater.cc:347] Connected to a master server at 127.14.68.62:40033
I20260927 10:55:54.031044 28833 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:54.031584 28833 heartbeater.cc:511] Master 127.14.68.62:40033 requested a full tablet report, sending...
I20260927 10:55:54.032733 28374 ts_manager.cc:194] Registered new tserver with Master: dfa9d318aaad4575bbf1900757eb8def (127.14.68.3:42187)
I20260927 10:55:54.033398 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:55:54.033419 28374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:60503
I20260927 10:55:54.043380 28414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:54.035572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41376 (local address 127.14.68.62:40033)
0927 10:55:54.035757 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:54.035763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:54.035938 (+ 175us) server_negotiation.cc:407] Connection header received
0927 10:55:54.036110 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:54.036113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:54.036158 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:54.036257 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:54.036965 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.037504 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.038415 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.038675 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.038796 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:54.040228 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:54.040232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:54.040235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:54.040260 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:54.041578 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:54.042056 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:54.042059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:54.042061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:54.042145 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:54.042441 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:54.042444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:54.042446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:54.042680 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:54.042833 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:54.043094 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 10:55:54.043211 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:55:54.046097 28374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:55:54.049166 28374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:55:54.063630 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:54.057234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:41709 (local address 127.0.0.1:41208)
0927 10:55:54.058016 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:54.058055 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:54.058195 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:54.059206 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:54.059212 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:54.059235 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:54.059374 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:54.059381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.060958 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.060961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:54.062072 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:54.062079 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.063071 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.063076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:54.063182 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:54.063224 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 10:55:54.063325 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 10:55:54.063442 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":2,"spinlock_wait_cycles":6528,"thread_start_us":76,"threads_started":1}
I20260927 10:55:54.063862 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:54.057817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40366 (local address 127.14.68.2:41787)
0927 10:55:54.058306 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:54.058322 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:54.058885 (+ 563us) server_negotiation.cc:407] Connection header received
0927 10:55:54.059125 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:54.059131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:54.059335 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:54.059470 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:54.059932 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.061089 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.062591 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.063218 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.063330 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:54.063463 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:54.063590 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 10:55:54.063718 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":143,"threads_started":1}
I20260927 10:55:54.064250 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:54.057668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:41787 (local address 127.0.0.1:40366)
0927 10:55:54.058775 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:54.058788 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:54.058973 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:54.059665 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:54.059670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:54.059688 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:54.059807 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:54.059813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.061228 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.061235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:54.062468 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:54.062479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.063345 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.063349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:54.063442 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:54.063468 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 10:55:54.063551 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 10:55:54.063619 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"mutex_wait_us":27,"thread_start_us":96,"threads_started":1}
I20260927 10:55:54.064272 28849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:54.058402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41208 (local address 127.14.68.1:41709)
0927 10:55:54.058728 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:54.058743 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:54.058762 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:55:54.058830 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:54.058835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:54.058996 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:54.059224 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:54.059492 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.060818 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.062198 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.062938 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.063029 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:54.063137 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:54.064040 (+ 903us) server_negotiation.cc:299] Negotiation successful
0927 10:55:54.064133 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":132,"threads_started":1}
I20260927 10:55:54.067235 28499 tablet_service.cc:1467] Processing CreateTablet for tablet 84b07268f71b499ba05ec6855e72d6eb (DEFAULT_TABLE table=test-table [id=92856e963aa54943826ef3222a44a827]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:54.067227 28633 tablet_service.cc:1467] Processing CreateTablet for tablet 84b07268f71b499ba05ec6855e72d6eb (DEFAULT_TABLE table=test-table [id=92856e963aa54943826ef3222a44a827]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:54.068220 28844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:54.057202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:42187 (local address 127.0.0.1:57024)
0927 10:55:54.058001 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:54.058048 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:54.058195 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:54.059415 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:54.059417 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:54.059456 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:54.059553 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:54.059559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.060656 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.060660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:54.062068 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:54.062079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.064294 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.064298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:54.067868 (+ 3570us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:54.067897 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 10:55:54.068005 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 10:55:54.068083 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":40,"spinlock_wait_cycles":6272,"thread_start_us":81,"threads_started":1}
I20260927 10:55:54.068368 28499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b07268f71b499ba05ec6855e72d6eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:54.068368 28633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b07268f71b499ba05ec6855e72d6eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:54.068533 28848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:54.058001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57024 (local address 127.14.68.3:42187)
0927 10:55:54.058635 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:54.058649 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:54.058678 (+ 29us) server_negotiation.cc:407] Connection header received
0927 10:55:54.058762 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:54.058768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:54.058997 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:54.059251 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:54.059727 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.060517 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.063193 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.064159 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.064287 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:54.064484 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:54.068040 (+ 3556us) server_negotiation.cc:299] Negotiation successful
0927 10:55:54.068150 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":143,"threads_started":1}
I20260927 10:55:54.071079 28767 tablet_service.cc:1467] Processing CreateTablet for tablet 84b07268f71b499ba05ec6855e72d6eb (DEFAULT_TABLE table=test-table [id=92856e963aa54943826ef3222a44a827]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:54.072007 28767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b07268f71b499ba05ec6855e72d6eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:54.075588 28851 tablet_bootstrap.cc:492] T 84b07268f71b499ba05ec6855e72d6eb P c6b943ae5b344a78941b00aa6af6089a: Bootstrap starting.
I20260927 10:55:54.076138 28850 tablet_bootstrap.cc:492] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87: Bootstrap starting.
I20260927 10:55:54.078635 28852 tablet_bootstrap.cc:492] T 84b07268f71b499ba05ec6855e72d6eb P dfa9d318aaad4575bbf1900757eb8def: Bootstrap starting.
I20260927 10:55:54.078697 28850 tablet_bootstrap.cc:654] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:54.080179 28850 log.cc:824] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:54.081159 28852 tablet_bootstrap.cc:654] T 84b07268f71b499ba05ec6855e72d6eb P dfa9d318aaad4575bbf1900757eb8def: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:54.081883 28852 log.cc:824] T 84b07268f71b499ba05ec6855e72d6eb P dfa9d318aaad4575bbf1900757eb8def: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:54.082007 28851 tablet_bootstrap.cc:654] T 84b07268f71b499ba05ec6855e72d6eb P c6b943ae5b344a78941b00aa6af6089a: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:54.082525 28850 tablet_bootstrap.cc:492] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87: No bootstrap required, opened a new log
I20260927 10:55:54.082737 28850 ts_tablet_manager.cc:1397] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 10:55:54.082870 28851 log.cc:824] T 84b07268f71b499ba05ec6855e72d6eb P c6b943ae5b344a78941b00aa6af6089a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:54.083858 28852 tablet_bootstrap.cc:492] T 84b07268f71b499ba05ec6855e72d6eb P dfa9d318aaad4575bbf1900757eb8def: No bootstrap required, opened a new log
I20260927 10:55:54.084059 28852 ts_tablet_manager.cc:1397] T 84b07268f71b499ba05ec6855e72d6eb P dfa9d318aaad4575bbf1900757eb8def: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:55:54.084858 28851 tablet_bootstrap.cc:492] T 84b07268f71b499ba05ec6855e72d6eb P c6b943ae5b344a78941b00aa6af6089a: No bootstrap required, opened a new log
I20260927 10:55:54.085037 28851 ts_tablet_manager.cc:1397] T 84b07268f71b499ba05ec6855e72d6eb P c6b943ae5b344a78941b00aa6af6089a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260927 10:55:54.086978 28850 raft_consensus.cc:348] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } }
I20260927 10:55:54.087263 28850 raft_consensus.cc:374] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:54.087169 28852 raft_consensus.cc:348] T 84b07268f71b499ba05ec6855e72d6eb P dfa9d318aaad4575bbf1900757eb8def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } }
I20260927 10:55:54.087352 28850 raft_consensus.cc:729] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8ef97f3e2a467b983129a63d0a3b87, State: Initialized, Role: FOLLOWER
I20260927 10:55:54.087423 28852 raft_consensus.cc:374] T 84b07268f71b499ba05ec6855e72d6eb P dfa9d318aaad4575bbf1900757eb8def [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:54.087484 28852 raft_consensus.cc:729] T 84b07268f71b499ba05ec6855e72d6eb P dfa9d318aaad4575bbf1900757eb8def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa9d318aaad4575bbf1900757eb8def, State: Initialized, Role: FOLLOWER
I20260927 10:55:54.087879 28850 consensus_queue.cc:260] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } }
I20260927 10:55:54.087916 28852 consensus_queue.cc:260] T 84b07268f71b499ba05ec6855e72d6eb P dfa9d318aaad4575bbf1900757eb8def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } }
I20260927 10:55:54.088768 28565 heartbeater.cc:503] Master 127.14.68.62:40033 was elected leader, sending a full tablet report...
I20260927 10:55:54.088903 28852 ts_tablet_manager.cc:1428] T 84b07268f71b499ba05ec6855e72d6eb P dfa9d318aaad4575bbf1900757eb8def: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:55:54.088912 28850 ts_tablet_manager.cc:1428] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:55:54.089041 28851 raft_consensus.cc:348] T 84b07268f71b499ba05ec6855e72d6eb P c6b943ae5b344a78941b00aa6af6089a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } }
I20260927 10:55:54.089337 28851 raft_consensus.cc:374] T 84b07268f71b499ba05ec6855e72d6eb P c6b943ae5b344a78941b00aa6af6089a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:54.089375 28833 heartbeater.cc:503] Master 127.14.68.62:40033 was elected leader, sending a full tablet report...
I20260927 10:55:54.089428 28851 raft_consensus.cc:729] T 84b07268f71b499ba05ec6855e72d6eb P c6b943ae5b344a78941b00aa6af6089a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b943ae5b344a78941b00aa6af6089a, State: Initialized, Role: FOLLOWER
I20260927 10:55:54.089845 28851 consensus_queue.cc:260] T 84b07268f71b499ba05ec6855e72d6eb P c6b943ae5b344a78941b00aa6af6089a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } }
I20260927 10:55:54.090778 28851 ts_tablet_manager.cc:1428] T 84b07268f71b499ba05ec6855e72d6eb P c6b943ae5b344a78941b00aa6af6089a: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260927 10:55:54.090945 28699 heartbeater.cc:503] Master 127.14.68.62:40033 was elected leader, sending a full tablet report...
W20260927 10:55:54.091848 28566 tablet.cc:2369] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:55:54.266289 28834 tablet.cc:2369] T 84b07268f71b499ba05ec6855e72d6eb P dfa9d318aaad4575bbf1900757eb8def: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:55:54.307672 28700 tablet.cc:2369] T 84b07268f71b499ba05ec6855e72d6eb P c6b943ae5b344a78941b00aa6af6089a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:55:54.330292 28857 raft_consensus.cc:482] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:55:54.330514 28857 raft_consensus.cc:504] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } }
I20260927 10:55:54.331802 28857 leader_election.cc:288] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa9d318aaad4575bbf1900757eb8def (127.14.68.3:42187), c6b943ae5b344a78941b00aa6af6089a (127.14.68.2:41787)
I20260927 10:55:54.335773 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:54.332171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:35397 (local address 127.14.68.2:41787)
0927 10:55:54.332303 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:54.332310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:54.332551 (+ 241us) server_negotiation.cc:407] Connection header received
0927 10:55:54.332706 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:54.332709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:54.332760 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:54.332848 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:54.333122 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.333605 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.334797 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.335192 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.335318 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:54.335379 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:54.335562 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 10:55:54.335631 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:55:54.335795 28859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:54.332067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:41787 (local address 127.14.68.1:35397)
0927 10:55:54.332448 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:54.332460 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:54.332587 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:54.332856 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:54.332859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:54.332874 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:54.333013 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:54.333019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.333728 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.333732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:54.334667 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:54.334677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.335322 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.335325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:54.335427 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:54.335449 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:55:54.335546 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 10:55:54.335616 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":26,"thread_start_us":140,"threads_started":1}
I20260927 10:55:54.335825 28848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:54.332170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:36477 (local address 127.14.68.3:42187)
0927 10:55:54.332306 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:54.332310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:54.332474 (+ 164us) server_negotiation.cc:407] Connection header received
0927 10:55:54.332634 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:54.332644 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:54.332687 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:54.332764 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:54.333035 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.333527 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.334799 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.335275 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.335386 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:54.335442 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:54.335622 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 10:55:54.335686 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:55:54.335845 28860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:54.332001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:42187 (local address 127.14.68.1:36477)
0927 10:55:54.332384 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:54.332395 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:54.332529 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:54.332786 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:54.332789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:54.332804 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:54.332933 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:54.332939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.333640 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.333643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:54.334684 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:54.334693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.335390 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.335398 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:54.335497 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:54.335523 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 10:55:54.335605 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 10:55:54.335680 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":1,"thread_start_us":111,"threads_started":1}
I20260927 10:55:54.336421 28787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b07268f71b499ba05ec6855e72d6eb" candidate_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa9d318aaad4575bbf1900757eb8def" is_pre_election: true
I20260927 10:55:54.336459 28653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b07268f71b499ba05ec6855e72d6eb" candidate_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b943ae5b344a78941b00aa6af6089a" is_pre_election: true
I20260927 10:55:54.336701 28787 raft_consensus.cc:2454] T 84b07268f71b499ba05ec6855e72d6eb P dfa9d318aaad4575bbf1900757eb8def [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f8ef97f3e2a467b983129a63d0a3b87 in term 0.
I20260927 10:55:54.336736 28653 raft_consensus.cc:2454] T 84b07268f71b499ba05ec6855e72d6eb P c6b943ae5b344a78941b00aa6af6089a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f8ef97f3e2a467b983129a63d0a3b87 in term 0.
I20260927 10:55:54.337133 28454 leader_election.cc:302] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8ef97f3e2a467b983129a63d0a3b87, dfa9d318aaad4575bbf1900757eb8def; no voters:
I20260927 10:55:54.337401 28857 raft_consensus.cc:2781] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:55:54.337482 28857 raft_consensus.cc:482] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:55:54.337523 28857 raft_consensus.cc:3037] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:54.338487 28857 raft_consensus.cc:504] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } }
I20260927 10:55:54.338814 28857 leader_election.cc:288] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 1 election: Requested vote from peers dfa9d318aaad4575bbf1900757eb8def (127.14.68.3:42187), c6b943ae5b344a78941b00aa6af6089a (127.14.68.2:41787)
I20260927 10:55:54.339092 28787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b07268f71b499ba05ec6855e72d6eb" candidate_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa9d318aaad4575bbf1900757eb8def"
I20260927 10:55:54.339170 28653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b07268f71b499ba05ec6855e72d6eb" candidate_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b943ae5b344a78941b00aa6af6089a"
I20260927 10:55:54.339255 28787 raft_consensus.cc:3037] T 84b07268f71b499ba05ec6855e72d6eb P dfa9d318aaad4575bbf1900757eb8def [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:54.339285 28653 raft_consensus.cc:3037] T 84b07268f71b499ba05ec6855e72d6eb P c6b943ae5b344a78941b00aa6af6089a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:54.340325 28653 raft_consensus.cc:2454] T 84b07268f71b499ba05ec6855e72d6eb P c6b943ae5b344a78941b00aa6af6089a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8ef97f3e2a467b983129a63d0a3b87 in term 1.
I20260927 10:55:54.340369 28787 raft_consensus.cc:2454] T 84b07268f71b499ba05ec6855e72d6eb P dfa9d318aaad4575bbf1900757eb8def [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8ef97f3e2a467b983129a63d0a3b87 in term 1.
I20260927 10:55:54.340586 28452 leader_election.cc:302] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8ef97f3e2a467b983129a63d0a3b87, c6b943ae5b344a78941b00aa6af6089a; no voters:
I20260927 10:55:54.340761 28857 raft_consensus.cc:2781] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:55:54.340981 28857 raft_consensus.cc:686] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 LEADER]: Becoming Leader. State: Replica: 4f8ef97f3e2a467b983129a63d0a3b87, State: Running, Role: LEADER
I20260927 10:55:54.341511 28857 consensus_queue.cc:237] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } }
I20260927 10:55:54.344712 28374 catalog_manager.cc:5495] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f8ef97f3e2a467b983129a63d0a3b87 (127.14.68.1). New cstate: current_term: 1 leader_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } health_report { overall_health: UNKNOWN } } }
I20260927 10:55:54.423769 28849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:54.420570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41214 (local address 127.14.68.1:41709)
0927 10:55:54.420699 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:54.420703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:54.420724 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:55:54.420799 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:54.420801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:54.420860 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:54.420948 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:55:54.421151 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.421638 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.422447 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.422655 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.422739 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:54.422994 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:55:54.423405 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:55:54.423492 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:54.423548 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 10:55:54.423598 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:55:54.429464 28787 raft_consensus.cc:1261] T 84b07268f71b499ba05ec6855e72d6eb P dfa9d318aaad4575bbf1900757eb8def [term 1 FOLLOWER]: Refusing update from remote peer 4f8ef97f3e2a467b983129a63d0a3b87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:55:54.429463 28653 raft_consensus.cc:1261] T 84b07268f71b499ba05ec6855e72d6eb P c6b943ae5b344a78941b00aa6af6089a [term 1 FOLLOWER]: Refusing update from remote peer 4f8ef97f3e2a467b983129a63d0a3b87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:55:54.430195 28861 consensus_queue.cc:1035] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:54.430387 28857 consensus_queue.cc:1035] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:54.437772 28865 mvcc.cc:205] Tried to move back new op lower bound from 7333914846936154112 to 7333914846585667584. Current Snapshot: MvccSnapshot[applied={T|T < 7333914846936154112}]
I20260927 10:55:54.446463 28372 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:55:54.446702 28372 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:55:54.449141 28372 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250: Sending DeleteTablet for 3 replicas of tablet 84b07268f71b499ba05ec6855e72d6eb
I20260927 10:55:54.449930 28499 tablet_service.cc:1514] Processing DeleteTablet for tablet 84b07268f71b499ba05ec6855e72d6eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:54 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:41208
I20260927 10:55:54.449967 28767 tablet_service.cc:1514] Processing DeleteTablet for tablet 84b07268f71b499ba05ec6855e72d6eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:54 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:57024
I20260927 10:55:54.450256 28633 tablet_service.cc:1514] Processing DeleteTablet for tablet 84b07268f71b499ba05ec6855e72d6eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:54 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:40366
I20260927 10:55:54.450464 28874 tablet_replica.cc:330] T 84b07268f71b499ba05ec6855e72d6eb P dfa9d318aaad4575bbf1900757eb8def: stopping tablet replica
I20260927 10:55:54.450706 28875 tablet_replica.cc:330] T 84b07268f71b499ba05ec6855e72d6eb P c6b943ae5b344a78941b00aa6af6089a: stopping tablet replica
I20260927 10:55:54.450796 28874 raft_consensus.cc:2229] T 84b07268f71b499ba05ec6855e72d6eb P dfa9d318aaad4575bbf1900757eb8def [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:55:54.450960 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 28436
I20260927 10:55:54.451157 28874 raft_consensus.cc:2258] T 84b07268f71b499ba05ec6855e72d6eb P dfa9d318aaad4575bbf1900757eb8def [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:54.451038 28875 raft_consensus.cc:2229] T 84b07268f71b499ba05ec6855e72d6eb P c6b943ae5b344a78941b00aa6af6089a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:55:54.452168 28875 raft_consensus.cc:2258] T 84b07268f71b499ba05ec6855e72d6eb P c6b943ae5b344a78941b00aa6af6089a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:54.452865 28874 ts_tablet_manager.cc:1905] T 84b07268f71b499ba05ec6855e72d6eb P dfa9d318aaad4575bbf1900757eb8def: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:54.453591 28875 ts_tablet_manager.cc:1905] T 84b07268f71b499ba05ec6855e72d6eb P c6b943ae5b344a78941b00aa6af6089a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:54.455866 28875 ts_tablet_manager.cc:1918] T 84b07268f71b499ba05ec6855e72d6eb P c6b943ae5b344a78941b00aa6af6089a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:55:54.455981 28875 log.cc:1196] T 84b07268f71b499ba05ec6855e72d6eb P c6b943ae5b344a78941b00aa6af6089a: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/84b07268f71b499ba05ec6855e72d6eb
I20260927 10:55:54.456288 28875 ts_tablet_manager.cc:1939] T 84b07268f71b499ba05ec6855e72d6eb P c6b943ae5b344a78941b00aa6af6089a: Deleting consensus metadata
I20260927 10:55:54.456449 28874 ts_tablet_manager.cc:1918] T 84b07268f71b499ba05ec6855e72d6eb P dfa9d318aaad4575bbf1900757eb8def: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:55:54.456558 28874 log.cc:1196] T 84b07268f71b499ba05ec6855e72d6eb P dfa9d318aaad4575bbf1900757eb8def: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/84b07268f71b499ba05ec6855e72d6eb
I20260927 10:55:54.456981 28874 ts_tablet_manager.cc:1939] T 84b07268f71b499ba05ec6855e72d6eb P dfa9d318aaad4575bbf1900757eb8def: Deleting consensus metadata
I20260927 10:55:54.457629 28358 catalog_manager.cc:4841] TS c6b943ae5b344a78941b00aa6af6089a (127.14.68.2:41787): tablet 84b07268f71b499ba05ec6855e72d6eb (table test-table [id=92856e963aa54943826ef3222a44a827]) successfully deleted
I20260927 10:55:54.458240 28360 catalog_manager.cc:4841] TS dfa9d318aaad4575bbf1900757eb8def (127.14.68.3:42187): tablet 84b07268f71b499ba05ec6855e72d6eb (table test-table [id=92856e963aa54943826ef3222a44a827]) successfully deleted
I20260927 10:55:54.459193 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 28570
W20260927 10:55:54.459177 28359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.68.1:41709 (error 108)
I20260927 10:55:54.460819 28844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:54.460571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:41709 (local address 127.0.0.1:41222)
0927 10:55:54.460687 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:54.460751 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.1:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 10:55:54.461067 28359 catalog_manager.cc:4584] TS 4f8ef97f3e2a467b983129a63d0a3b87 (127.14.68.1:41709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 84b07268f71b499ba05ec6855e72d6eb: Network error: Client connection negotiation failed: client connection to 127.14.68.1:41709: connect: Connection refused (error 111)
I20260927 10:55:54.465076 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 28704
I20260927 10:55:54.470400 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 28342
I20260927 10:55:54.476749 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:40033
--webserver_interface=127.14.68.62
--webserver_port=46313
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:40033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:54.584501 28877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:54.584744 28877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:54.584784 28877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:54.588793 28877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:55:54.588842 28877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:54.588874 28877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:54.588891 28877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:55:54.588953 28877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:55:54.594064 28877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:40033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:40033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=46313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:54.595036 28877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:54.596055 28877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:54.608403 28883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:54.608425 28884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:54.608438 28886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:54.608726 28877 server_base.cc:1034] running on GCE node
I20260927 10:55:54.609272 28877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:54.610338 28877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:54.611532 28877 hybrid_clock.cc:648] HybridClock initialized: now 1790506554611517 us; error 31 us; skew 500 ppm
Sep 27 10:55:54 dist-test-slave-tz51 krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506554, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:54.614274 28877 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:55:54.615096 28877 webserver.cc:460] Webserver started at http://127.14.68.62:46313/ using document root <none> and password file <none>
I20260927 10:55:54.615658 28877 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:54.615794 28877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:54.619459 28877 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 10:55:54.621505 28893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:54.622735 28877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:55:54.622879 28877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "b6f6b89832ff479c89587f0802e98250"
format_stamp: "Formatted at 2026-09-27 10:55:53 on dist-test-slave-tz51"
server_key: "fc34ea30f323a52b6d049a8b5fa995d7"
server_key_iv: "3150a5635f4248f605ebe05d3db2d104"
server_key_version: "encryptionkey@0"
I20260927 10:55:54.623449 28877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:54.639990 28877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:54.642756 28877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:54.642946 28877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:54.649588 28877 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:40033
I20260927 10:55:54.649866 28945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:40033 every 8 connection(s)
I20260927 10:55:54.650667 28877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:55:54.652632 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 28877
I20260927 10:55:54.653053 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:41709
--local_ip_for_outbound_sockets=127.14.68.1
--tserver_master_addrs=127.14.68.62:40033
--webserver_port=35745
--webserver_interface=127.14.68.1
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:55:54.655367 28946 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:55:54.657142 28946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250: Bootstrap starting.
I20260927 10:55:54.673633 28946 log.cc:824] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:54.682080 28946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:55:54.682452 28946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250: Bootstrap complete.
I20260927 10:55:54.686501 28946 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f6b89832ff479c89587f0802e98250" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40033 } }
I20260927 10:55:54.686899 28946 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6f6b89832ff479c89587f0802e98250, State: Initialized, Role: FOLLOWER
I20260927 10:55:54.687428 28946 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f6b89832ff479c89587f0802e98250" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40033 } }
I20260927 10:55:54.687542 28946 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:55:54.687634 28946 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:55:54.687793 28946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:55:54.689615 28946 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f6b89832ff479c89587f0802e98250" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40033 } }
I20260927 10:55:54.689951 28946 leader_election.cc:302] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6f6b89832ff479c89587f0802e98250; no voters:
I20260927 10:55:54.690299 28946 leader_election.cc:288] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 10:55:54.690356 28950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 10:55:54.690557 28950 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 2 LEADER]: Becoming Leader. State: Replica: b6f6b89832ff479c89587f0802e98250, State: Running, Role: LEADER
I20260927 10:55:54.690838 28950 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f6b89832ff479c89587f0802e98250" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40033 } }
I20260927 10:55:54.691494 28946 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:55:54.691828 28952 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b6f6b89832ff479c89587f0802e98250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f6b89832ff479c89587f0802e98250" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40033 } } }
I20260927 10:55:54.691944 28952 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:54.692190 28951 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6f6b89832ff479c89587f0802e98250. Latest consensus state: current_term: 2 leader_uuid: "b6f6b89832ff479c89587f0802e98250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f6b89832ff479c89587f0802e98250" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40033 } } }
I20260927 10:55:54.692315 28951 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:54.692723 28954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:55:54.697556 28954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:55:54.698169 28954 catalog_manager.cc:1259] Loaded cluster ID: ae10e8997f71481fa546527580d14ff9
I20260927 10:55:54.698242 28954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:55:54.699645 28954 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 10:55:54.704526 28954 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 10:55:54.709852 28954 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:55:54.710551 28954 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250: Loaded TSK: 0
I20260927 10:55:54.711679 28954 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:55:54.765256 28948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:54.765530 28948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:54.765604 28948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:54.769896 28948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:54.769980 28948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:54.770071 28948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:55:54.775286 28948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:41709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=35745
--tserver_master_addrs=127.14.68.62:40033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:54.776360 28948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:54.777356 28948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:54.787771 28973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:54.787813 28974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:54.787813 28976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:54.788039 28948 server_base.cc:1034] running on GCE node
I20260927 10:55:54.788476 28948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:54.789147 28948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:54.790329 28948 hybrid_clock.cc:648] HybridClock initialized: now 1790506554790305 us; error 42 us; skew 500 ppm
Sep 27 10:55:54 dist-test-slave-tz51 krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506554, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:54.793272 28948 init.cc:377] Logged in from keytab as kudu/127.14.68.1@KRBTEST.COM (short username kudu)
I20260927 10:55:54.794193 28948 webserver.cc:460] Webserver started at http://127.14.68.1:35745/ using document root <none> and password file <none>
I20260927 10:55:54.794729 28948 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:54.794831 28948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:54.798595 28948 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:54.800949 28983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:54.802202 28948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:55:54.802332 28948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "4f8ef97f3e2a467b983129a63d0a3b87"
format_stamp: "Formatted at 2026-09-27 10:55:53 on dist-test-slave-tz51"
server_key: "3fc191b38edd11e56b4ff999d0ae1bfd"
server_key_iv: "180b0882db05b9aba0535396986da20e"
server_key_version: "encryptionkey@0"
I20260927 10:55:54.802867 28948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:54.814447 28948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:54.818607 28948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:54.818794 28948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:54.819466 28948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:54.820737 28990 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 10:55:54.825784 28948 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 10:55:54.825871 28948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260927 10:55:54.825958 28948 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 10:55:54.827862 28948 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 10:55:54.827929 28948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:54.828225 28990 tablet_bootstrap.cc:492] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87: Bootstrap starting.
I20260927 10:55:54.838577 28948 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:41709
I20260927 10:55:54.839534 28948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:55:54.841104 29097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:41709 every 8 connection(s)
I20260927 10:55:54.849776 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 28948
I20260927 10:55:54.850273 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:41787
--local_ip_for_outbound_sockets=127.14.68.2
--tserver_master_addrs=127.14.68.62:40033
--webserver_port=38983
--webserver_interface=127.14.68.2
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:55:54 dist-test-slave-tz51 krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506554, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:55:54.851594 28990 log.cc:824] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:54.858326 29101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:54.841737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.1:50169)
0927 10:55:54.842463 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:54.842516 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:54.843411 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:54.844569 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:54.844577 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:54.845009 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:54.845474 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:54.845488 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.846847 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.846850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:54.847771 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:54.847788 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.848532 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.848536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:54.848733 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:54.849421 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:54.849438 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:54.852352 (+ 2914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:54.854956 (+ 2604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:54.854964 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:54.854980 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:54.855304 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:54.855745 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:54.855748 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:54.855749 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:54.855909 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:54.856645 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:54.856651 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:54.856908 (+ 257us) client_negotiation.cc:769] Sending connection context
0927 10:55:54.857077 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 10:55:54.857385 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":156,"threads_started":1}
I20260927 10:55:54.860168 29100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:54.841709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:50169 (local address 127.14.68.62:40033)
0927 10:55:54.842090 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:54.842113 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:54.842630 (+ 517us) server_negotiation.cc:407] Connection header received
0927 10:55:54.843857 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:54.843869 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:54.844174 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:54.844518 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:54.845664 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.846665 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.847899 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.848377 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.848562 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:54.852559 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:54.852566 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:54.852585 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:54.852631 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:54.854812 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:54.855437 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:54.855443 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:54.855448 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:54.855588 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:54.856018 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:54.856021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:54.856023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:54.856458 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:54.856583 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:54.858799 (+ 2216us) server_negotiation.cc:299] Negotiation successful
0927 10:55:54.859171 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":107,"threads_started":1}
I20260927 10:55:54.860836 28990 tablet_bootstrap.cc:492] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:55:54.861274 28990 tablet_bootstrap.cc:492] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87: Bootstrap complete.
I20260927 10:55:54.862409 29098 heartbeater.cc:347] Connected to a master server at 127.14.68.62:40033
I20260927 10:55:54.862567 28990 ts_tablet_manager.cc:1397] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87: Time spent bootstrapping tablet: real 0.035s user 0.013s sys 0.016s
I20260927 10:55:54.862664 29098 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:54.863247 29098 heartbeater.cc:511] Master 127.14.68.62:40033 requested a full tablet report, sending...
I20260927 10:55:54.864820 28990 raft_consensus.cc:348] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } }
I20260927 10:55:54.865352 28990 raft_consensus.cc:729] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8ef97f3e2a467b983129a63d0a3b87, State: Initialized, Role: FOLLOWER
I20260927 10:55:54.865804 28990 consensus_queue.cc:260] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } }
I20260927 10:55:54.865875 28910 ts_manager.cc:194] Registered new tserver with Master: 4f8ef97f3e2a467b983129a63d0a3b87 (127.14.68.1:41709)
I20260927 10:55:54.866847 28990 ts_tablet_manager.cc:1428] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260927 10:55:54.869710 28910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:50169
I20260927 10:55:54.871831 29098 heartbeater.cc:503] Master 127.14.68.62:40033 was elected leader, sending a full tablet report...
Sep 27 10:55:54 dist-test-slave-tz51 krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506554, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for kudu/127.14.68.1@KRBTEST.COM
I20260927 10:55:54.878960 29105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:54.868783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:41709 (local address 127.0.0.1:41234)
0927 10:55:54.869168 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:54.869223 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:54.869337 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:54.870012 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:54.870018 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:54.870429 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:54.870550 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:54.870558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.872039 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.872042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:54.872770 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:54.872779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.873404 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.873408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:54.873523 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:54.874052 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:54.874063 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:54.875537 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:54.877310 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:54.877320 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:54.877325 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:54.877541 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:54.877859 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:54.877863 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:54.877865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:54.877948 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:54.878367 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:54.878373 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:54.878491 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 10:55:54.878628 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 10:55:54.878799 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":103,"threads_started":1}
I20260927 10:55:54.878998 29106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:54.868851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.14.68.1:41709)
0927 10:55:54.869581 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:54.869592 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:54.869612 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:55:54.869670 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:54.869675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:54.869859 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:54.869993 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:54.870689 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.871872 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.872890 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:54.873286 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:54.873381 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:54.875676 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:54.875691 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:54.875744 (+ 53us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:54.875773 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:54.877197 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:54.877690 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:54.877694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:54.877696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:54.877767 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:54.878045 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:54.878048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:54.878050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:54.878258 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:54.878388 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:54.878708 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 10:55:54.878859 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":116,"threads_started":1}
I20260927 10:55:54.879951 29014 tablet_service.cc:1514] Processing DeleteTablet for tablet 84b07268f71b499ba05ec6855e72d6eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:54 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:41234
I20260927 10:55:54.880712 29107 tablet_replica.cc:330] stopping tablet replica
I20260927 10:55:54.880977 29107 raft_consensus.cc:2229] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:55:54.881201 29107 raft_consensus.cc:2258] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:54.882069 29107 ts_tablet_manager.cc:1905] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:54.885427 29107 ts_tablet_manager.cc:1918] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:55:54.885502 29107 log.cc:1196] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/wal/wals/84b07268f71b499ba05ec6855e72d6eb
I20260927 10:55:54.885762 29107 ts_tablet_manager.cc:1939] T 84b07268f71b499ba05ec6855e72d6eb P 4f8ef97f3e2a467b983129a63d0a3b87: Deleting consensus metadata
I20260927 10:55:54.887249 28895 catalog_manager.cc:4841] TS 4f8ef97f3e2a467b983129a63d0a3b87 (127.14.68.1:41709): tablet 84b07268f71b499ba05ec6855e72d6eb (table test-table [id=92856e963aa54943826ef3222a44a827]) successfully deleted
W20260927 10:55:54.967301 29102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:54.967577 29102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:54.967659 29102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:54.971578 29102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:54.971657 29102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:54.971766 29102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:55:54.976835 29102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:41787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=38983
--tserver_master_addrs=127.14.68.62:40033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:54.977862 29102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:54.978844 29102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:54.988278 29115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:54.988351 29102 server_base.cc:1034] running on GCE node
W20260927 10:55:54.988303 29112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:54.988301 29113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:54.988857 29102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:54.989496 29102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:54.990685 29102 hybrid_clock.cc:648] HybridClock initialized: now 1790506554990668 us; error 34 us; skew 500 ppm
Sep 27 10:55:54 dist-test-slave-tz51 krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506554, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:54.993512 29102 init.cc:377] Logged in from keytab as kudu/127.14.68.2@KRBTEST.COM (short username kudu)
I20260927 10:55:54.994299 29102 webserver.cc:460] Webserver started at http://127.14.68.2:38983/ using document root <none> and password file <none>
I20260927 10:55:54.994836 29102 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:54.994943 29102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:54.998667 29102 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:55.000643 29122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:55.001827 29102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:55.001981 29102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "c6b943ae5b344a78941b00aa6af6089a"
format_stamp: "Formatted at 2026-09-27 10:55:53 on dist-test-slave-tz51"
server_key: "19c5ec5b23544aa567062eadc500fe3f"
server_key_iv: "d2b0ddb3fb0485ff8c8481d1271ea80b"
server_key_version: "encryptionkey@0"
I20260927 10:55:55.002502 29102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:55.026116 29102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:55.029286 29102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:55.029491 29102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:55.030109 29102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:55.030932 29102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:55.031003 29102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:55.031098 29102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:55.031148 29102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:55.040241 29102 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:41787
I20260927 10:55:55.040274 29235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:41787 every 8 connection(s)
I20260927 10:55:55.041198 29102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:55:55.047425 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 29102
I20260927 10:55:55.047892 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:42187
--local_ip_for_outbound_sockets=127.14.68.3
--tserver_master_addrs=127.14.68.62:40033
--webserver_port=45513
--webserver_interface=127.14.68.3
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:55:55 dist-test-slave-tz51 krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506554, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:55:55.056197 29100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:55.042923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:56397 (local address 127.14.68.62:40033)
0927 10:55:55.043080 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:55.043085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:55.043996 (+ 911us) server_negotiation.cc:407] Connection header received
0927 10:55:55.045283 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:55.045286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:55.045338 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:55.045412 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:55.046552 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.047068 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.048622 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.049060 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.049195 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:55.052456 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:55.052459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:55.052462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:55.052490 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:55.054055 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:55.054592 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:55.054595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:55.054596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:55.054670 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:55.055016 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:55.055019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:55.055021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:55.055212 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:55.055320 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:55.055865 (+ 545us) server_negotiation.cc:299] Negotiation successful
0927 10:55:55.056008 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:55:55.057097 29238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:55.043365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.2:56397)
0927 10:55:55.043851 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:55.043893 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:55.045090 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:55.045588 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:55.045595 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:55.045934 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:55.046422 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:55.046436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.047237 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.047240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:55.048489 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:55.048500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.049269 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.049276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:55.049524 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:55.050199 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:55.050225 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:55.052268 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:55.054179 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:55.054185 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:55.054199 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:55.054465 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:55.054764 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:55.054769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:55.054772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:55.054910 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:55.055317 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:55.055324 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:55.055592 (+ 268us) client_negotiation.cc:769] Sending connection context
0927 10:55:55.055806 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 10:55:55.056151 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":101,"threads_started":1}
I20260927 10:55:55.058516 29236 heartbeater.cc:347] Connected to a master server at 127.14.68.62:40033
I20260927 10:55:55.058791 29236 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:55.059271 29236 heartbeater.cc:511] Master 127.14.68.62:40033 requested a full tablet report, sending...
I20260927 10:55:55.060406 28910 ts_manager.cc:194] Registered new tserver with Master: c6b943ae5b344a78941b00aa6af6089a (127.14.68.2:41787)
I20260927 10:55:55.061146 28910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:56397
W20260927 10:55:55.158798 29239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:55.159026 29239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:55.159065 29239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:55.162978 29239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:55.163029 29239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:55.163084 29239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:55:55.168020 29239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:42187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=45513
--tserver_master_addrs=127.14.68.62:40033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:55.168889 29239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:55.169770 29239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:55.178910 29244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:55.178918 29245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:55.179100 29239 server_base.cc:1034] running on GCE node
W20260927 10:55:55.178985 29247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:55.179531 29239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:55.180132 29239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:55.181288 29239 hybrid_clock.cc:648] HybridClock initialized: now 1790506555181256 us; error 43 us; skew 500 ppm
Sep 27 10:55:55 dist-test-slave-tz51 krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506555, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:55.183921 29239 init.cc:377] Logged in from keytab as kudu/127.14.68.3@KRBTEST.COM (short username kudu)
I20260927 10:55:55.184932 29239 webserver.cc:460] Webserver started at http://127.14.68.3:45513/ using document root <none> and password file <none>
I20260927 10:55:55.185454 29239 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:55.185544 29239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:55.189213 29239 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:55:55.191076 29254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:55.192361 29239 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:55.192492 29239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "dfa9d318aaad4575bbf1900757eb8def"
format_stamp: "Formatted at 2026-09-27 10:55:53 on dist-test-slave-tz51"
server_key: "4bad641643f49e925265423519e5564d"
server_key_iv: "8e986a96f39020ed6ada88c97098496f"
server_key_version: "encryptionkey@0"
I20260927 10:55:55.193081 29239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:55.204644 29239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:55.207805 29239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:55.207981 29239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:55.208526 29239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:55.209321 29239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:55.209389 29239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:55.209462 29239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:55.209509 29239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:55.218557 29239 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:42187
I20260927 10:55:55.218627 29367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:42187 every 8 connection(s)
I20260927 10:55:55.219547 29239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:55:55.224978 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 29239
Sep 27 10:55:55 dist-test-slave-tz51 krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506555, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:55:55.234295 29372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:55.225622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41380 (local address 127.14.68.62:40033)
0927 10:55:55.225880 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:55.225884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:55.226023 (+ 139us) server_negotiation.cc:407] Connection header received
0927 10:55:55.226208 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:55.226211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:55.226260 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:55.226352 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:55.227160 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.227933 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.228883 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.229175 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.229260 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:55.230719 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:55.230722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:55.230724 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:55.230778 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:55.232622 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:55.233080 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:55.233083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:55.233084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:55.233146 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:55.233406 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:55.233410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:55.233412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:55.233592 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:55.233733 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:55.233957 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 10:55:55.234111 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":197,"thread_start_us":89,"threads_started":1}
I20260927 10:55:55.234393 29100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:55.221332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:59007 (local address 127.14.68.62:40033)
0927 10:55:55.221464 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:55.221468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:55.222306 (+ 838us) server_negotiation.cc:407] Connection header received
0927 10:55:55.223660 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:55.223663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:55.223752 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:55.223854 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:55.224978 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.225712 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.226963 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.227294 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.227373 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:55.230265 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:55.230268 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:55.230270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:55.230290 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:55.232270 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:55.232774 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:55.232776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:55.232778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:55.232845 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:55.233239 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:55.233242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:55.233244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:55.233471 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:55.233563 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:55.234142 (+ 579us) server_negotiation.cc:299] Negotiation successful
0927 10:55:55.234240 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":53}
I20260927 10:55:55.235409 29370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:55.221727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.3:59007)
0927 10:55:55.222166 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:55.222206 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:55.223440 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:55.224037 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:55.224049 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:55.224384 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:55.224835 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:55.224848 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.225862 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.225865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:55.226854 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:55.226864 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.227432 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.227435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:55.227639 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:55.228394 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:55.228410 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:55.230092 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:55.232384 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:55.232388 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:55.232399 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:55.232645 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:55.232937 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:55.232942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:55.232947 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:55.233127 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:55.233582 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:55.233587 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:55.233902 (+ 315us) client_negotiation.cc:769] Sending connection context
0927 10:55:55.234104 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 10:55:55.234460 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":115,"threads_started":1}
I20260927 10:55:55.236522 29368 heartbeater.cc:347] Connected to a master server at 127.14.68.62:40033
I20260927 10:55:55.236750 29368 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:55.237195 29368 heartbeater.cc:511] Master 127.14.68.62:40033 requested a full tablet report, sending...
I20260927 10:55:55.238183 28909 ts_manager.cc:194] Registered new tserver with Master: dfa9d318aaad4575bbf1900757eb8def (127.14.68.3:42187)
I20260927 10:55:55.238722 28909 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:59007
I20260927 10:55:55.247933 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:55:55.252379 29100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:55.248562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41386 (local address 127.14.68.62:40033)
0927 10:55:55.248750 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:55.248755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:55.248913 (+ 158us) server_negotiation.cc:407] Connection header received
0927 10:55:55.249101 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:55.249103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:55.249203 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:55.249315 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:55:55.249560 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.250058 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.250950 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.251178 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.251294 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:55.251601 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:55:55.252022 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:55:55.252161 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:55.252197 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 10:55:55.252245 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 10:55:55.252987 28909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:55:55.256186 28909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:55:55.263187 29014 tablet_service.cc:1467] Processing CreateTablet for tablet 151bcb4c5d3e4a7ba312b9d9ebce3324 (DEFAULT_TABLE table=test-table [id=a7fe8d23c32e4ced8c0c323c309a8502]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:55.263847 29014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 151bcb4c5d3e4a7ba312b9d9ebce3324. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:55.266777 28990 tablet_bootstrap.cc:492] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87: Bootstrap starting.
I20260927 10:55:55.267342 29105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:55.262086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:42187 (local address 127.0.0.1:57036)
0927 10:55:55.262232 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:55.262251 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:55.262335 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:55.263624 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:55.263627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:55.263643 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:55.263765 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:55.263772 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.265032 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.265035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:55.265817 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:55.265825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.266941 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.266944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:55.267039 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:55.267087 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 10:55:55.267135 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 10:55:55.267225 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 10:55:55.267613 28990 tablet_bootstrap.cc:654] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:55.267980 29378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:55.262522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57036 (local address 127.14.68.3:42187)
0927 10:55:55.263140 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:55.263152 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:55.263175 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:55:55.263252 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:55.263258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:55.263461 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:55.263644 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:55.263898 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.264885 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.265980 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.266809 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.267536 (+ 727us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:55.267700 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:55.267790 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 10:55:55.267873 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":114,"threads_started":1}
I20260927 10:55:55.268697 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:55.262429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:41787 (local address 127.0.0.1:40370)
0927 10:55:55.262687 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:55.262701 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:55.262783 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:55.264412 (+ 1629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:55.264415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:55.264434 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:55.264513 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:55.264518 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.265959 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.265963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:55.267071 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:55.267079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.268233 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.268238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:55.268337 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:55.268356 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 10:55:55.268461 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 10:55:55.268543 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":94,"threads_started":1}
I20260927 10:55:55.268721 29379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:55.262848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.14.68.2:41787)
0927 10:55:55.263977 (+ 1129us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:55.263988 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:55.264009 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:55:55.264079 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:55.264084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:55.264262 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:55.264418 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:55.264638 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.265820 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.267193 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.268054 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.268222 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:55.268358 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:55.268497 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 10:55:55.268581 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1031,"thread_start_us":113,"threads_started":1}
I20260927 10:55:55.269353 28990 tablet_bootstrap.cc:492] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87: No bootstrap required, opened a new log
I20260927 10:55:55.269459 28990 ts_tablet_manager.cc:1397] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 10:55:55.269869 28990 raft_consensus.cc:348] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } }
I20260927 10:55:55.269986 28990 raft_consensus.cc:374] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:55.270038 28990 raft_consensus.cc:729] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8ef97f3e2a467b983129a63d0a3b87, State: Initialized, Role: FOLLOWER
I20260927 10:55:55.270201 28990 consensus_queue.cc:260] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } }
I20260927 10:55:55.270478 28990 ts_tablet_manager.cc:1428] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:55:55.271515 29302 tablet_service.cc:1467] Processing CreateTablet for tablet 151bcb4c5d3e4a7ba312b9d9ebce3324 (DEFAULT_TABLE table=test-table [id=a7fe8d23c32e4ced8c0c323c309a8502]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:55.272644 29302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 151bcb4c5d3e4a7ba312b9d9ebce3324. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:55.272267 29170 tablet_service.cc:1467] Processing CreateTablet for tablet 151bcb4c5d3e4a7ba312b9d9ebce3324 (DEFAULT_TABLE table=test-table [id=a7fe8d23c32e4ced8c0c323c309a8502]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:55.273229 29170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 151bcb4c5d3e4a7ba312b9d9ebce3324. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:55.279161 29381 tablet_bootstrap.cc:492] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def: Bootstrap starting.
I20260927 10:55:55.279215 29382 tablet_bootstrap.cc:492] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P c6b943ae5b344a78941b00aa6af6089a: Bootstrap starting.
I20260927 10:55:55.281407 29382 tablet_bootstrap.cc:654] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P c6b943ae5b344a78941b00aa6af6089a: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:55.281443 29381 tablet_bootstrap.cc:654] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:55.282140 29382 log.cc:824] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P c6b943ae5b344a78941b00aa6af6089a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:55.282179 29381 log.cc:824] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:55.284049 29382 tablet_bootstrap.cc:492] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P c6b943ae5b344a78941b00aa6af6089a: No bootstrap required, opened a new log
I20260927 10:55:55.284063 29381 tablet_bootstrap.cc:492] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def: No bootstrap required, opened a new log
I20260927 10:55:55.284240 29382 ts_tablet_manager.cc:1397] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P c6b943ae5b344a78941b00aa6af6089a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:55:55.284260 29381 ts_tablet_manager.cc:1397] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:55:55.287132 29381 raft_consensus.cc:348] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } }
I20260927 10:55:55.287141 29382 raft_consensus.cc:348] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P c6b943ae5b344a78941b00aa6af6089a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } }
I20260927 10:55:55.287359 29381 raft_consensus.cc:374] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:55.287382 29382 raft_consensus.cc:374] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P c6b943ae5b344a78941b00aa6af6089a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:55.287426 29381 raft_consensus.cc:729] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa9d318aaad4575bbf1900757eb8def, State: Initialized, Role: FOLLOWER
I20260927 10:55:55.287442 29382 raft_consensus.cc:729] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P c6b943ae5b344a78941b00aa6af6089a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b943ae5b344a78941b00aa6af6089a, State: Initialized, Role: FOLLOWER
I20260927 10:55:55.287869 29382 consensus_queue.cc:260] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P c6b943ae5b344a78941b00aa6af6089a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } }
I20260927 10:55:55.287869 29381 consensus_queue.cc:260] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } }
I20260927 10:55:55.288480 29236 heartbeater.cc:503] Master 127.14.68.62:40033 was elected leader, sending a full tablet report...
I20260927 10:55:55.288483 29368 heartbeater.cc:503] Master 127.14.68.62:40033 was elected leader, sending a full tablet report...
I20260927 10:55:55.288832 29382 ts_tablet_manager.cc:1428] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P c6b943ae5b344a78941b00aa6af6089a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:55:55.288832 29381 ts_tablet_manager.cc:1428] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260927 10:55:55.292079 29237 tablet.cc:2369] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P c6b943ae5b344a78941b00aa6af6089a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:55:55.340942 29099 tablet.cc:2369] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:55:55.446825 29386 raft_consensus.cc:482] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:55:55.447064 29386 raft_consensus.cc:504] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } }
I20260927 10:55:55.448354 29386 leader_election.cc:288] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6b943ae5b344a78941b00aa6af6089a (127.14.68.2:41787), 4f8ef97f3e2a467b983129a63d0a3b87 (127.14.68.1:41709)
I20260927 10:55:55.451910 29379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:55.448731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:57223 (local address 127.14.68.2:41787)
0927 10:55:55.448854 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:55.448857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:55.448872 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:55:55.448961 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:55.448964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:55.449019 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:55.449106 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:55.449402 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.449904 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.450930 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.451335 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.451455 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:55.451515 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:55.451723 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 10:55:55.451790 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:55:55.451910 29370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:55.448610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:41787 (local address 127.14.68.3:57223)
0927 10:55:55.448743 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:55.448759 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:55.448856 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:55.449124 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:55.449127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:55.449147 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:55.449280 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:55.449286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.450023 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.450027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:55.450801 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:55.450809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.451456 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.451460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:55.451571 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:55.451588 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 10:55:55.451686 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 10:55:55.451778 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 10:55:55.452580 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:55.448751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:35161 (local address 127.14.68.1:41709)
0927 10:55:55.449049 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:55.449055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:55.449348 (+ 293us) server_negotiation.cc:407] Connection header received
0927 10:55:55.449498 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:55.449501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:55.449569 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:55.449686 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:55.449954 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.450456 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.451489 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.452049 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.452178 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:55.452352 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:55.452394 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 10:55:55.452465 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":105,"threads_started":1}
I20260927 10:55:55.452600 29388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:55.448625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:41709 (local address 127.14.68.3:35161)
0927 10:55:55.449248 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:55.449259 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:55.449377 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:55.449697 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:55.449699 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:55.449716 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:55.449834 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:55.449840 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.450587 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.450591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:55.451358 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:55.451366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.452206 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.452210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:55.452298 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:55.452318 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:55:55.452384 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 10:55:55.452448 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":311,"thread_start_us":113,"threads_started":1}
I20260927 10:55:55.452574 29190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "151bcb4c5d3e4a7ba312b9d9ebce3324" candidate_uuid: "dfa9d318aaad4575bbf1900757eb8def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b943ae5b344a78941b00aa6af6089a" is_pre_election: true
I20260927 10:55:55.452944 29190 raft_consensus.cc:2454] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P c6b943ae5b344a78941b00aa6af6089a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa9d318aaad4575bbf1900757eb8def in term 0.
I20260927 10:55:55.453233 29048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "151bcb4c5d3e4a7ba312b9d9ebce3324" candidate_uuid: "dfa9d318aaad4575bbf1900757eb8def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" is_pre_election: true
I20260927 10:55:55.453459 29255 leader_election.cc:302] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6b943ae5b344a78941b00aa6af6089a, dfa9d318aaad4575bbf1900757eb8def; no voters:
I20260927 10:55:55.453505 29048 raft_consensus.cc:2454] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa9d318aaad4575bbf1900757eb8def in term 0.
I20260927 10:55:55.453756 29386 raft_consensus.cc:2781] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:55:55.453874 29386 raft_consensus.cc:482] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:55:55.453923 29386 raft_consensus.cc:3037] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:55.454960 29386 raft_consensus.cc:504] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } }
I20260927 10:55:55.455294 29386 leader_election.cc:288] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def [CANDIDATE]: Term 1 election: Requested vote from peers c6b943ae5b344a78941b00aa6af6089a (127.14.68.2:41787), 4f8ef97f3e2a467b983129a63d0a3b87 (127.14.68.1:41709)
I20260927 10:55:55.455590 29190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "151bcb4c5d3e4a7ba312b9d9ebce3324" candidate_uuid: "dfa9d318aaad4575bbf1900757eb8def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b943ae5b344a78941b00aa6af6089a"
I20260927 10:55:55.455641 29048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "151bcb4c5d3e4a7ba312b9d9ebce3324" candidate_uuid: "dfa9d318aaad4575bbf1900757eb8def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8ef97f3e2a467b983129a63d0a3b87"
I20260927 10:55:55.455749 29190 raft_consensus.cc:3037] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P c6b943ae5b344a78941b00aa6af6089a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:55.455832 29048 raft_consensus.cc:3037] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:55.456799 29190 raft_consensus.cc:2454] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P c6b943ae5b344a78941b00aa6af6089a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa9d318aaad4575bbf1900757eb8def in term 1.
I20260927 10:55:55.456799 29048 raft_consensus.cc:2454] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa9d318aaad4575bbf1900757eb8def in term 1.
I20260927 10:55:55.457055 29255 leader_election.cc:302] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6b943ae5b344a78941b00aa6af6089a, dfa9d318aaad4575bbf1900757eb8def; no voters:
I20260927 10:55:55.457230 29386 raft_consensus.cc:2781] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:55:55.457458 29386 raft_consensus.cc:686] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def [term 1 LEADER]: Becoming Leader. State: Replica: dfa9d318aaad4575bbf1900757eb8def, State: Running, Role: LEADER
I20260927 10:55:55.457808 29386 consensus_queue.cc:237] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } }
I20260927 10:55:55.461370 28909 catalog_manager.cc:5495] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa9d318aaad4575bbf1900757eb8def (127.14.68.3). New cstate: current_term: 1 leader_uuid: "dfa9d318aaad4575bbf1900757eb8def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } health_report { overall_health: UNKNOWN } } }
W20260927 10:55:55.470480 29369 tablet.cc:2369] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:55:55.515213 29378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:55.511772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57040 (local address 127.14.68.3:42187)
0927 10:55:55.511913 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:55.511920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:55.511937 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:55:55.512030 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:55.512032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:55.512085 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:55.512196 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:55:55.512448 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.512950 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.513780 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:55.514001 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:55.514107 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:55.514429 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:55:55.514832 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:55:55.514967 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:55.515008 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 10:55:55.515061 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:55:55.520799 29190 raft_consensus.cc:1261] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P c6b943ae5b344a78941b00aa6af6089a [term 1 FOLLOWER]: Refusing update from remote peer dfa9d318aaad4575bbf1900757eb8def: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:55:55.520797 29048 raft_consensus.cc:1261] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Refusing update from remote peer dfa9d318aaad4575bbf1900757eb8def: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:55:55.521513 29389 consensus_queue.cc:1035] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:55.521690 29386 consensus_queue.cc:1035] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:55.536365 28909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:55:55.536533 28909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:55:55.538793 28909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250: Sending DeleteTablet for 3 replicas of tablet 151bcb4c5d3e4a7ba312b9d9ebce3324
I20260927 10:55:55.539502 29170 tablet_service.cc:1514] Processing DeleteTablet for tablet 151bcb4c5d3e4a7ba312b9d9ebce3324 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:55 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:40370
I20260927 10:55:55.539527 29302 tablet_service.cc:1514] Processing DeleteTablet for tablet 151bcb4c5d3e4a7ba312b9d9ebce3324 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:55 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:57036
I20260927 10:55:55.539932 29014 tablet_service.cc:1514] Processing DeleteTablet for tablet 151bcb4c5d3e4a7ba312b9d9ebce3324 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:55 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:41234
I20260927 10:55:55.539965 29402 tablet_replica.cc:330] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P c6b943ae5b344a78941b00aa6af6089a: stopping tablet replica
I20260927 10:55:55.540266 29404 tablet_replica.cc:330] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87: stopping tablet replica
I20260927 10:55:55.540297 29402 raft_consensus.cc:2229] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P c6b943ae5b344a78941b00aa6af6089a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:55:55.540455 29404 raft_consensus.cc:2229] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:55:55.540587 29402 raft_consensus.cc:2258] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P c6b943ae5b344a78941b00aa6af6089a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:55.540604 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 28948
I20260927 10:55:55.542037 29403 tablet_replica.cc:330] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def: stopping tablet replica
I20260927 10:55:55.542270 29402 ts_tablet_manager.cc:1905] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P c6b943ae5b344a78941b00aa6af6089a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:55.542306 29403 raft_consensus.cc:2229] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:55:55.542766 29403 raft_consensus.cc:2258] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:55.545655 29402 ts_tablet_manager.cc:1918] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P c6b943ae5b344a78941b00aa6af6089a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:55:55.545747 29402 log.cc:1196] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P c6b943ae5b344a78941b00aa6af6089a: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/151bcb4c5d3e4a7ba312b9d9ebce3324
I20260927 10:55:55.546082 29402 ts_tablet_manager.cc:1939] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P c6b943ae5b344a78941b00aa6af6089a: Deleting consensus metadata
I20260927 10:55:55.546303 29403 ts_tablet_manager.cc:1905] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:55.547297 28894 catalog_manager.cc:4841] TS c6b943ae5b344a78941b00aa6af6089a (127.14.68.2:41787): tablet 151bcb4c5d3e4a7ba312b9d9ebce3324 (table test-table [id=a7fe8d23c32e4ced8c0c323c309a8502]) successfully deleted
I20260927 10:55:55.548869 29403 ts_tablet_manager.cc:1918] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:55:55.548960 29403 log.cc:1196] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/151bcb4c5d3e4a7ba312b9d9ebce3324
I20260927 10:55:55.549275 29403 ts_tablet_manager.cc:1939] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P dfa9d318aaad4575bbf1900757eb8def: Deleting consensus metadata
I20260927 10:55:55.550362 28896 catalog_manager.cc:4841] TS dfa9d318aaad4575bbf1900757eb8def (127.14.68.3:42187): tablet 151bcb4c5d3e4a7ba312b9d9ebce3324 (table test-table [id=a7fe8d23c32e4ced8c0c323c309a8502]) successfully deleted
W20260927 10:55:55.551234 28895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.68.1:41709 (error 108)
I20260927 10:55:55.551786 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 29102
I20260927 10:55:55.556622 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:55.556331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:41709 (local address 127.0.0.1:41248)
0927 10:55:55.556474 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:55.556544 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.1:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 10:55:55.556828 28895 catalog_manager.cc:4584] TS 4f8ef97f3e2a467b983129a63d0a3b87 (127.14.68.1:41709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 151bcb4c5d3e4a7ba312b9d9ebce3324: Network error: Client connection negotiation failed: client connection to 127.14.68.1:41709: connect: Connection refused (error 111)
I20260927 10:55:55.559427 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 29239
I20260927 10:55:55.564277 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 28877
I20260927 10:55:55.570411 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:40033
--webserver_interface=127.14.68.62
--webserver_port=46313
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:40033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:55.677723 29406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:55.677984 29406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:55.678022 29406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:55.681998 29406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:55:55.682054 29406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:55.682073 29406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:55.682089 29406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:55:55.682104 29406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:55:55.687317 29406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:40033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:40033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=46313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:55.688408 29406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:55.689453 29406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:55.697834 29412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:55.697885 29406 server_base.cc:1034] running on GCE node
W20260927 10:55:55.697825 29411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:55.697837 29414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:55.698546 29406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:55.699609 29406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:55.700794 29406 hybrid_clock.cc:648] HybridClock initialized: now 1790506555700772 us; error 35 us; skew 500 ppm
Sep 27 10:55:55 dist-test-slave-tz51 krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506555, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:55.708639 29406 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:55:55.709607 29406 webserver.cc:460] Webserver started at http://127.14.68.62:46313/ using document root <none> and password file <none>
I20260927 10:55:55.710170 29406 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:55.710261 29406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:55.714054 29406 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20260927 10:55:55.716192 29421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:55.717358 29406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:55.717491 29406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "b6f6b89832ff479c89587f0802e98250"
format_stamp: "Formatted at 2026-09-27 10:55:53 on dist-test-slave-tz51"
server_key: "fc34ea30f323a52b6d049a8b5fa995d7"
server_key_iv: "3150a5635f4248f605ebe05d3db2d104"
server_key_version: "encryptionkey@0"
I20260927 10:55:55.718025 29406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:55.743834 29406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:55.746711 29406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:55.746907 29406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:55.754042 29406 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:40033
I20260927 10:55:55.754335 29473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:40033 every 8 connection(s)
I20260927 10:55:55.755132 29406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:55:55.756644 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 29406
I20260927 10:55:55.757114 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:41709
--local_ip_for_outbound_sockets=127.14.68.1
--tserver_master_addrs=127.14.68.62:40033
--webserver_port=35745
--webserver_interface=127.14.68.1
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:55:55.760241 29474 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:55:55.761924 29474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250: Bootstrap starting.
I20260927 10:55:55.778826 29474 log.cc:824] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:55.790104 29474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:55:55.790479 29474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250: Bootstrap complete.
I20260927 10:55:55.794616 29474 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f6b89832ff479c89587f0802e98250" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40033 } }
I20260927 10:55:55.794981 29474 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6f6b89832ff479c89587f0802e98250, State: Initialized, Role: FOLLOWER
I20260927 10:55:55.795452 29474 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f6b89832ff479c89587f0802e98250" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40033 } }
I20260927 10:55:55.795570 29474 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:55:55.795665 29474 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:55:55.795794 29474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 2 FOLLOWER]: Advancing to term 3
I20260927 10:55:55.797585 29474 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f6b89832ff479c89587f0802e98250" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40033 } }
I20260927 10:55:55.797907 29474 leader_election.cc:302] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6f6b89832ff479c89587f0802e98250; no voters:
I20260927 10:55:55.798219 29474 leader_election.cc:288] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 10:55:55.798298 29478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 10:55:55.798491 29478 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [term 3 LEADER]: Becoming Leader. State: Replica: b6f6b89832ff479c89587f0802e98250, State: Running, Role: LEADER
I20260927 10:55:55.798781 29478 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f6b89832ff479c89587f0802e98250" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40033 } }
I20260927 10:55:55.799428 29474 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:55:55.799667 29479 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6f6b89832ff479c89587f0802e98250. Latest consensus state: current_term: 3 leader_uuid: "b6f6b89832ff479c89587f0802e98250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f6b89832ff479c89587f0802e98250" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40033 } } }
I20260927 10:55:55.800397 29479 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:55.800292 29480 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b6f6b89832ff479c89587f0802e98250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f6b89832ff479c89587f0802e98250" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 40033 } } }
I20260927 10:55:55.800721 29480 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6f6b89832ff479c89587f0802e98250 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:55.801000 29488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:55:55.805783 29488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:55:55.806492 29488 catalog_manager.cc:1259] Loaded cluster ID: ae10e8997f71481fa546527580d14ff9
I20260927 10:55:55.806551 29488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 10:55:55.807471 29488 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb462c1e9ad google::LogMessage::Fail() at ??:0
@ 0x7fb462c2188a google::LogMessage::SendToLog() at ??:0
@ 0x7fb462c1e4de google::LogMessage::Flush() at ??:0
@ 0x7fb462c1fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb46d55dfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fb46d55ed2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb46d55b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fb46d59f640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb464e703d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb464e70c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fb464e723f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb464e61512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb4664a56db start_thread at ??:0
@ 0x7fb46171c61f clone at ??:0
W20260927 10:55:55.883195 29476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:55.883460 29476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:55.883575 29476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:55.887530 29476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:55.887622 29476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:55.887750 29476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:55:55.893036 29476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:41709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=35745
--tserver_master_addrs=127.14.68.62:40033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:55.894069 29476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:55.895062 29476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:55.904422 29516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:55.904423 29515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:55.904422 29518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:55.904779 29476 server_base.cc:1034] running on GCE node
I20260927 10:55:55.905239 29476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:55.906075 29476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:55.907277 29476 hybrid_clock.cc:648] HybridClock initialized: now 1790506555907245 us; error 41 us; skew 500 ppm
Sep 27 10:55:55 dist-test-slave-tz51 krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506555, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:55.910509 29476 init.cc:377] Logged in from keytab as kudu/127.14.68.1@KRBTEST.COM (short username kudu)
I20260927 10:55:55.911315 29476 webserver.cc:460] Webserver started at http://127.14.68.1:35745/ using document root <none> and password file <none>
I20260927 10:55:55.911938 29476 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:55.912001 29476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:55.915768 29476 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 10:55:55.917901 29525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:55.919152 29476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:55.919277 29476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "4f8ef97f3e2a467b983129a63d0a3b87"
format_stamp: "Formatted at 2026-09-27 10:55:53 on dist-test-slave-tz51"
server_key: "3fc191b38edd11e56b4ff999d0ae1bfd"
server_key_iv: "180b0882db05b9aba0535396986da20e"
server_key_version: "encryptionkey@0"
I20260927 10:55:55.919852 29476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:55.937565 29476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:55.941107 29476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:55.941331 29476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:55.941988 29476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:55.943185 29532 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 10:55:55.948475 29476 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 10:55:55.948552 29476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 10:55:55.948654 29476 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 10:55:55.950325 29476 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 10:55:55.950397 29476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 10:55:55.950762 29532 tablet_bootstrap.cc:492] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87: Bootstrap starting.
I20260927 10:55:55.961181 29476 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:41709
I20260927 10:55:55.961398 29639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:41709 every 8 connection(s)
I20260927 10:55:55.962149 29476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:55:55.963966 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 29476
I20260927 10:55:55.964442 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:41787
--local_ip_for_outbound_sockets=127.14.68.2
--tserver_master_addrs=127.14.68.62:40033
--webserver_port=38983
--webserver_interface=127.14.68.2
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:55:55.972816 29532 log.cc:824] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:55.981966 29532 tablet_bootstrap.cc:492] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:55:55.982396 29532 tablet_bootstrap.cc:492] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87: Bootstrap complete.
I20260927 10:55:55.983887 29532 ts_tablet_manager.cc:1397] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.007s
I20260927 10:55:55.987280 29532 raft_consensus.cc:348] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } }
I20260927 10:55:55.987825 29532 raft_consensus.cc:729] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8ef97f3e2a467b983129a63d0a3b87, State: Initialized, Role: FOLLOWER
I20260927 10:55:55.988440 29532 consensus_queue.cc:260] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } }
I20260927 10:55:55.989487 29532 ts_tablet_manager.cc:1428] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260927 10:55:56.056782 29642 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:55.964254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.1:36595)
0927 10:55:55.965628 (+ 1374us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:55.965688 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:55.966911 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:56.055813 (+ 88902us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1123,"thread_start_us":152,"threads_started":1}
W20260927 10:55:56.057456 29640 heartbeater.cc:644] Failed to heartbeat to 127.14.68.62:40033 (0 consecutive failures): Network error: Failed to ping master at 127.14.68.62:40033: Client connection negotiation failed: client connection to 127.14.68.62:40033: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 10:55:56.058198 29642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:56.057908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.1:54873)
0927 10:55:56.058054 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:56.058126 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 10:55:56.058873 29642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:56.058682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.1:54069)
0927 10:55:56.058787 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:56.058828 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 10:55:56.059113 29640 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:55:56.077697 29643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:56.077975 29643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:56.078081 29643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:56.082021 29643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:56.082089 29643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:56.082167 29643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:55:56.087353 29643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:41787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=38983
--tserver_master_addrs=127.14.68.62:40033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:56.088371 29643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:56.089382 29643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:55:56.099323 29643 server_base.cc:1034] running on GCE node
W20260927 10:55:56.099270 29651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:56.099296 29653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:56.099254 29650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:56.099963 29643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:56.100548 29643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:56.101715 29643 hybrid_clock.cc:648] HybridClock initialized: now 1790506556101692 us; error 40 us; skew 500 ppm
Sep 27 10:55:56 dist-test-slave-tz51 krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506556, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:56.104337 29643 init.cc:377] Logged in from keytab as kudu/127.14.68.2@KRBTEST.COM (short username kudu)
I20260927 10:55:56.105134 29643 webserver.cc:460] Webserver started at http://127.14.68.2:38983/ using document root <none> and password file <none>
I20260927 10:55:56.105666 29643 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:56.105763 29643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:56.109360 29643 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:55:56.111296 29660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:56.112447 29643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:56.112584 29643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "c6b943ae5b344a78941b00aa6af6089a"
format_stamp: "Formatted at 2026-09-27 10:55:53 on dist-test-slave-tz51"
server_key: "19c5ec5b23544aa567062eadc500fe3f"
server_key_iv: "d2b0ddb3fb0485ff8c8481d1271ea80b"
server_key_version: "encryptionkey@0"
I20260927 10:55:56.113129 29643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:56.139907 29643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:56.142573 29643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:56.142767 29643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:56.143327 29643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:56.144114 29643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:56.144184 29643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:56.144248 29643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:56.144302 29643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:56.153023 29643 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:41787
I20260927 10:55:56.153041 29773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:41787 every 8 connection(s)
I20260927 10:55:56.153963 29643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:55:56.157332 29776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:56.155869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.2:49023)
0927 10:55:56.156282 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:56.156660 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":96,"threads_started":1}
W20260927 10:55:56.157733 29774 heartbeater.cc:644] Failed to heartbeat to 127.14.68.62:40033 (0 consecutive failures): Network error: Failed to ping master at 127.14.68.62:40033: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
I20260927 10:55:56.158282 29776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:56.158073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.2:60951)
0927 10:55:56.158174 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:56.158222 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 10:55:56.158842 29776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:56.158649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.2:54633)
0927 10:55:56.158741 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:56.158785 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 10:55:56.159005 29774 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 10:55:56.161505 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 29643
I20260927 10:55:56.161896 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:42187
--local_ip_for_outbound_sockets=127.14.68.3
--tserver_master_addrs=127.14.68.62:40033
--webserver_port=45513
--webserver_interface=127.14.68.3
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:56.270983 29777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:56.271220 29777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:56.271262 29777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:56.275241 29777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:56.275295 29777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:56.275398 29777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:55:56.280612 29777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:42187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=45513
--tserver_master_addrs=127.14.68.62:40033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:55:56.281564 29777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:56.282531 29777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:56.291782 29785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:56.291760 29783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:56.291837 29777 server_base.cc:1034] running on GCE node
W20260927 10:55:56.291764 29782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:56.292318 29777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:56.292917 29777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:56.294116 29777 hybrid_clock.cc:648] HybridClock initialized: now 1790506556294104 us; error 31 us; skew 500 ppm
Sep 27 10:55:56 dist-test-slave-tz51 krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506556, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:56.296952 29777 init.cc:377] Logged in from keytab as kudu/127.14.68.3@KRBTEST.COM (short username kudu)
I20260927 10:55:56.297833 29777 webserver.cc:460] Webserver started at http://127.14.68.3:45513/ using document root <none> and password file <none>
I20260927 10:55:56.298377 29777 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:56.298465 29777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:56.302242 29777 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:56.304312 29792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:56.305604 29777 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:56.305732 29777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "dfa9d318aaad4575bbf1900757eb8def"
format_stamp: "Formatted at 2026-09-27 10:55:53 on dist-test-slave-tz51"
server_key: "4bad641643f49e925265423519e5564d"
server_key_iv: "8e986a96f39020ed6ada88c97098496f"
server_key_version: "encryptionkey@0"
I20260927 10:55:56.306293 29777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:56.322238 29777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:56.325565 29777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:56.325752 29777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:56.326349 29777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:56.327149 29777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:56.327219 29777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:56.327288 29777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:56.327323 29777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:56.336349 29777 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:42187
I20260927 10:55:56.336374 29905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:42187 every 8 connection(s)
I20260927 10:55:56.337278 29777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:55:56.337677 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 29777
I20260927 10:55:56.341027 29909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:56.339470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.3:51969)
0927 10:55:56.339974 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:56.340362 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":110,"threads_started":1}
W20260927 10:55:56.341460 29906 heartbeater.cc:644] Failed to heartbeat to 127.14.68.62:40033 (0 consecutive failures): Network error: Failed to ping master at 127.14.68.62:40033: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
I20260927 10:55:56.341966 29909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:56.341784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.3:56801)
0927 10:55:56.341876 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:56.341918 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 10:55:56.342549 29909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:56.342346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.3:42495)
0927 10:55:56.342441 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:56.342490 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 10:55:56.342850 29906 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 10:55:57.060163 29910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:57.059740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.1:51983)
0927 10:55:57.060002 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:57.060058 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
I20260927 10:55:57.160027 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:57.159595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.2:48489)
0927 10:55:57.159883 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:57.159935 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":101,"threads_started":1}
I20260927 10:55:57.343890 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:57.343439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.3:55445)
0927 10:55:57.343679 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:57.343791 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260927 10:55:57.844997 29913 raft_consensus.cc:482] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:55:57.845184 29913 raft_consensus.cc:504] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } }
I20260927 10:55:57.846323 29913 leader_election.cc:288] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa9d318aaad4575bbf1900757eb8def (127.14.68.3:42187), c6b943ae5b344a78941b00aa6af6089a (127.14.68.2:41787)
Sep 27 10:55:57 dist-test-slave-tz51 krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506555, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.3@KRBTEST.COM
Sep 27 10:55:57 dist-test-slave-tz51 krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506555, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.2@KRBTEST.COM
I20260927 10:55:57.860446 29915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:57.846539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:41787 (local address 127.14.68.1:40043)
0927 10:55:57.847155 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:57.847169 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:57.847283 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:57.849397 (+ 2114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:57.849399 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:57.849571 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:57.849777 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:57.849791 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:57.851051 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:57.851059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:57.852051 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:57.852060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:57.852762 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:57.852765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:57.852962 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:57.853587 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:57.853632 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:57.855556 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:57.857742 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:57.857745 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:57.857747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:57.858042 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:57.858375 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:57.858378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:57.858380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:57.858556 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:57.859303 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:57.859309 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:57.859663 (+ 354us) client_negotiation.cc:769] Sending connection context
0927 10:55:57.859900 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 10:55:57.860244 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":202,"spinlock_wait_cycles":2176,"thread_start_us":122,"threads_started":1}
I20260927 10:55:57.860498 29917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:57.846813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:40043 (local address 127.14.68.2:41787)
0927 10:55:57.847275 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:57.847287 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:57.847322 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:55:57.848133 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:57.848141 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:57.848592 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:57.848919 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:57.849952 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:57.850900 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:57.852172 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:57.852628 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:57.852813 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:57.855754 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:57.855763 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:57.855781 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:57.855813 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:57.857647 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:57.858154 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:57.858159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:57.858165 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:57.858278 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:57.858692 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:57.858695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:57.858697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:57.859180 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:57.859319 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:57.860048 (+ 729us) server_negotiation.cc:299] Negotiation successful
0927 10:55:57.860323 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":72,"threads_started":1}
I20260927 10:55:57.860497 29916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:57.846679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:51933 (local address 127.14.68.3:42187)
0927 10:55:57.847038 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:57.847055 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:57.847103 (+ 48us) server_negotiation.cc:407] Connection header received
0927 10:55:57.848134 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:57.848141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:57.848592 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:57.848919 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:57.849944 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:57.850908 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:57.852166 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:57.852601 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:57.852750 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:57.855432 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:57.855437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:57.855448 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:57.855473 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:57.857553 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:57.858288 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:57.858292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:57.858296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:57.858401 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:57.858694 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:57.858697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:57.858698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:57.859180 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:57.859320 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:57.860048 (+ 728us) server_negotiation.cc:299] Negotiation successful
0927 10:55:57.860323 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":82,"threads_started":1}
I20260927 10:55:57.860447 29914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:57.846515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:42187 (local address 127.14.68.1:51933)
0927 10:55:57.846891 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:57.846907 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:57.847013 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:57.848967 (+ 1954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:57.848973 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:57.849333 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:57.849777 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:57.849792 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:57.851048 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:57.851051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:57.852050 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:57.852060 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:57.852724 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:57.852728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:57.852971 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:57.853651 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:57.853662 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:57.855202 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:57.857676 (+ 2474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:57.857682 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:57.857695 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:57.858075 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:57.858482 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:57.858485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:57.858487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:57.858555 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:57.859303 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:57.859309 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:57.859663 (+ 354us) client_negotiation.cc:769] Sending connection context
0927 10:55:57.859900 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 10:55:57.860292 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":267,"spinlock_wait_cycles":16640,"thread_start_us":76,"threads_started":1}
W20260927 10:55:57.862866 29526 leader_election.cc:341] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c6b943ae5b344a78941b00aa6af6089a (127.14.68.2:41787): Not found: Tablet not found: 151bcb4c5d3e4a7ba312b9d9ebce3324
W20260927 10:55:57.863056 29528 leader_election.cc:341] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfa9d318aaad4575bbf1900757eb8def (127.14.68.3:42187): Not found: Tablet not found: 151bcb4c5d3e4a7ba312b9d9ebce3324
I20260927 10:55:57.863153 29528 leader_election.cc:302] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f8ef97f3e2a467b983129a63d0a3b87; no voters: c6b943ae5b344a78941b00aa6af6089a, dfa9d318aaad4575bbf1900757eb8def
I20260927 10:55:57.863420 29913 raft_consensus.cc:2726] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:55:58.061158 29914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:58.060856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.1:47477)
0927 10:55:58.060974 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:58.061035 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 10:55:58.161087 29918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:58.160714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.2:45767)
0927 10:55:58.160935 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:58.160985 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":68,"threads_started":1}
I20260927 10:55:58.345000 29919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:58.344594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.3:50371)
0927 10:55:58.344835 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:58.344887 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":69,"threads_started":1}
I20260927 10:55:59.062320 29920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:59.061874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.1:47365)
0927 10:55:59.062169 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:59.062228 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":83,"threads_started":1}
I20260927 10:55:59.162192 29921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:59.161804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.2:41471)
0927 10:55:59.162049 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:59.162109 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
I20260927 10:55:59.346010 29922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:59.345634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.3:56135)
0927 10:55:59.345871 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:59.345923 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":80,"threads_started":1}
I20260927 10:55:59.495942 29923 raft_consensus.cc:482] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:55:59.496100 29923 raft_consensus.cc:504] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } }
I20260927 10:55:59.496552 29923 leader_election.cc:288] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa9d318aaad4575bbf1900757eb8def (127.14.68.3:42187), c6b943ae5b344a78941b00aa6af6089a (127.14.68.2:41787)
W20260927 10:55:59.497345 29528 leader_election.cc:341] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfa9d318aaad4575bbf1900757eb8def (127.14.68.3:42187): Not found: Tablet not found: 151bcb4c5d3e4a7ba312b9d9ebce3324
W20260927 10:55:59.497483 29526 leader_election.cc:341] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c6b943ae5b344a78941b00aa6af6089a (127.14.68.2:41787): Not found: Tablet not found: 151bcb4c5d3e4a7ba312b9d9ebce3324
I20260927 10:55:59.497632 29526 leader_election.cc:302] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f8ef97f3e2a467b983129a63d0a3b87; no voters: c6b943ae5b344a78941b00aa6af6089a, dfa9d318aaad4575bbf1900757eb8def
I20260927 10:55:59.497797 29923 raft_consensus.cc:2726] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:56:00.063536 29924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:00.063134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.1:34291)
0927 10:56:00.063394 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:00.063445 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":97,"threads_started":1}
I20260927 10:56:00.163442 29925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:00.162987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.2:46731)
0927 10:56:00.163268 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:00.163330 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":77,"threads_started":1}
I20260927 10:56:00.347257 29926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:00.346857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.3:39639)
0927 10:56:00.347104 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:00.347169 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20260927 10:56:01.064980 29927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:01.064524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.1:33153)
0927 10:56:01.064831 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:01.064888 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":76,"threads_started":1}
I20260927 10:56:01.164497 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:01.164110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.2:33929)
0927 10:56:01.164361 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:01.164413 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":84,"threads_started":1}
I20260927 10:56:01.217204 29929 raft_consensus.cc:482] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:56:01.217365 29929 raft_consensus.cc:504] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } }
I20260927 10:56:01.217826 29929 leader_election.cc:288] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa9d318aaad4575bbf1900757eb8def (127.14.68.3:42187), c6b943ae5b344a78941b00aa6af6089a (127.14.68.2:41787)
W20260927 10:56:01.218518 29526 leader_election.cc:341] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c6b943ae5b344a78941b00aa6af6089a (127.14.68.2:41787): Not found: Tablet not found: 151bcb4c5d3e4a7ba312b9d9ebce3324
W20260927 10:56:01.218629 29528 leader_election.cc:341] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfa9d318aaad4575bbf1900757eb8def (127.14.68.3:42187): Not found: Tablet not found: 151bcb4c5d3e4a7ba312b9d9ebce3324
I20260927 10:56:01.218739 29528 leader_election.cc:302] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f8ef97f3e2a467b983129a63d0a3b87; no voters: c6b943ae5b344a78941b00aa6af6089a, dfa9d318aaad4575bbf1900757eb8def
I20260927 10:56:01.218904 29929 raft_consensus.cc:2726] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:56:01.348502 29930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:01.348078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.3:37219)
0927 10:56:01.348362 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:01.348415 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":123,"threads_started":1}
I20260927 10:56:02.066272 29931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:02.065828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.1:58593)
0927 10:56:02.066124 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:02.066179 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260927 10:56:02.165565 29932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:02.165186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.2:60655)
0927 10:56:02.165421 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:02.165473 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":82,"threads_started":1}
I20260927 10:56:02.349545 29933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:02.349171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.3:59741)
0927 10:56:02.349403 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:02.349452 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":86,"threads_started":1}
I20260927 10:56:03.067395 29934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:03.067004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.1:50389)
0927 10:56:03.067253 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:03.067304 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
I20260927 10:56:03.083950 29935 raft_consensus.cc:482] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:56:03.084065 29935 raft_consensus.cc:504] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } }
I20260927 10:56:03.084499 29935 leader_election.cc:288] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa9d318aaad4575bbf1900757eb8def (127.14.68.3:42187), c6b943ae5b344a78941b00aa6af6089a (127.14.68.2:41787)
W20260927 10:56:03.085178 29528 leader_election.cc:341] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfa9d318aaad4575bbf1900757eb8def (127.14.68.3:42187): Not found: Tablet not found: 151bcb4c5d3e4a7ba312b9d9ebce3324
W20260927 10:56:03.085331 29526 leader_election.cc:341] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c6b943ae5b344a78941b00aa6af6089a (127.14.68.2:41787): Not found: Tablet not found: 151bcb4c5d3e4a7ba312b9d9ebce3324
I20260927 10:56:03.085422 29526 leader_election.cc:302] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f8ef97f3e2a467b983129a63d0a3b87; no voters: c6b943ae5b344a78941b00aa6af6089a, dfa9d318aaad4575bbf1900757eb8def
I20260927 10:56:03.085567 29935 raft_consensus.cc:2726] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:56:03.166661 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:03.166256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.2:45387)
0927 10:56:03.166516 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:03.166568 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":71,"threads_started":1}
I20260927 10:56:03.350589 29937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:03.350181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.3:35235)
0927 10:56:03.350448 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:03.350501 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":70,"threads_started":1}
I20260927 10:56:04.068594 29938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:04.068173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.1:56233)
0927 10:56:04.068445 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:04.068498 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":94,"threads_started":1}
I20260927 10:56:04.167876 29939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:04.167413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.2:49197)
0927 10:56:04.167666 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:04.167780 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":98,"threads_started":1}
I20260927 10:56:04.351832 29940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:04.351347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.3:51647)
0927 10:56:04.351641 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:04.351730 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":87,"threads_started":1}
I20260927 10:56:05.069867 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:05.069443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.1:45509)
0927 10:56:05.069701 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:05.069766 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":74,"threads_started":1}
I20260927 10:56:05.169029 29942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:05.168593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.2:39701)
0927 10:56:05.168871 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:05.168932 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":94,"threads_started":1}
I20260927 10:56:05.352955 29943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:05.352528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.3:38213)
0927 10:56:05.352806 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:05.352863 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":99,"threads_started":1}
I20260927 10:56:06.071066 29944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:06.070674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.1:43925)
0927 10:56:06.070917 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:06.070973 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260927 10:56:06.170298 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:06.169834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.2:48109)
0927 10:56:06.170103 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:06.170179 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":96,"threads_started":1}
I20260927 10:56:06.354171 29946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:06.353768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.3:39543)
0927 10:56:06.354022 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:06.354077 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":88,"threads_started":1}
I20260927 10:56:07.072330 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:07.071923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.1:58947)
0927 10:56:07.072175 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:07.072235 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
I20260927 10:56:07.171567 29948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:07.171120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.2:55367)
0927 10:56:07.171405 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:07.171469 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":89,"threads_started":1}
I20260927 10:56:07.355353 29949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:07.354897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.3:53011)
0927 10:56:07.355173 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:07.355261 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":87,"threads_started":1}
I20260927 10:56:08.073457 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:08.073058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.1:56201)
0927 10:56:08.073313 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:08.073365 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":90,"threads_started":1}
I20260927 10:56:08.172902 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:08.172412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.2:59151)
0927 10:56:08.172731 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:08.172800 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":81,"threads_started":1}
I20260927 10:56:08.356642 29952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:08.356185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.3:36423)
0927 10:56:08.356485 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:08.356542 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":126,"threads_started":1}
I20260927 10:56:09.074656 29953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:09.074233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.1:39991)
0927 10:56:09.074498 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:09.074561 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":87,"threads_started":1}
I20260927 10:56:09.174147 29954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:09.173704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.2:53193)
0927 10:56:09.173966 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:09.174044 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
I20260927 10:56:09.357962 29955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:09.357536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.3:52143)
0927 10:56:09.357816 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:09.357868 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":89,"threads_started":1}
I20260927 10:56:09.723420 29956 raft_consensus.cc:482] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:56:09.723551 29956 raft_consensus.cc:504] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa9d318aaad4575bbf1900757eb8def" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42187 } } peers { permanent_uuid: "c6b943ae5b344a78941b00aa6af6089a" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 41787 } } peers { permanent_uuid: "4f8ef97f3e2a467b983129a63d0a3b87" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41709 } }
I20260927 10:56:09.724026 29956 leader_election.cc:288] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa9d318aaad4575bbf1900757eb8def (127.14.68.3:42187), c6b943ae5b344a78941b00aa6af6089a (127.14.68.2:41787)
W20260927 10:56:09.724831 29526 leader_election.cc:341] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c6b943ae5b344a78941b00aa6af6089a (127.14.68.2:41787): Not found: Tablet not found: 151bcb4c5d3e4a7ba312b9d9ebce3324
W20260927 10:56:09.724941 29528 leader_election.cc:341] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfa9d318aaad4575bbf1900757eb8def (127.14.68.3:42187): Not found: Tablet not found: 151bcb4c5d3e4a7ba312b9d9ebce3324
I20260927 10:56:09.725061 29528 leader_election.cc:302] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f8ef97f3e2a467b983129a63d0a3b87; no voters: c6b943ae5b344a78941b00aa6af6089a, dfa9d318aaad4575bbf1900757eb8def
I20260927 10:56:09.725219 29956 raft_consensus.cc:2726] T 151bcb4c5d3e4a7ba312b9d9ebce3324 P 4f8ef97f3e2a467b983129a63d0a3b87 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:56:10.075942 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:10.075360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.1:35865)
0927 10:56:10.075647 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:10.075799 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":73,"threads_started":1}
I20260927 10:56:10.175365 29958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:10.174954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.2:54453)
0927 10:56:10.175221 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:10.175272 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":77,"threads_started":1}
I20260927 10:56:10.359144 29959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:10.358731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.3:58831)
0927 10:56:10.358993 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:10.359045 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":99,"threads_started":1}
I20260927 10:56:11.077315 29960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:11.076877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.1:40247)
0927 10:56:11.077142 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:11.077199 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
I20260927 10:56:11.176785 29961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:11.176332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:40033 (local address 127.14.68.2:48985)
0927 10:56:11.176618 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:11.176686 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":100,"threads_started":1}
I20260927 10:56:11.344026 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 29476
I20260927 10:56:11.349799 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 29643
I20260927 10:56:11.354836 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 29777
2026-09-27T10:56:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19394 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:56:11 dist-test-slave-tz51 krb5kdc[29964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:56:11 dist-test-slave-tz51 krb5kdc[29964](info): set up 2 sockets
Sep 27 10:56:11 dist-test-slave-tz51 krb5kdc[29964](info): commencing operation
krb5kdc: starting...
W20260927 10:56:12.392156 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:56:12 dist-test-slave-tz51 krb5kdc[29964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:56:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:56:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:56:12.515137 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:44105
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:41573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:44105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:56:12.623087 29977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:56:12.623375 29977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:56:12.623453 29977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:56:12.627470 29977 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 10:56:12.627540 29977 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 10:56:12.627566 29977 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 10:56:12.627616 29977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:56:12.627656 29977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:56:12.627720 29977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:56:12.627758 29977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:56:12.627791 29977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:56:12.632846 29977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:41573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:44105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:44105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:56:12.633792 29977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:56:12.634773 29977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:56:12.643067 29985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:56:12.643095 29977 server_base.cc:1034] running on GCE node
W20260927 10:56:12.643101 29982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:12.643105 29983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:56:12.643953 29977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:56:12.644985 29977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:56:12.646154 29977 hybrid_clock.cc:648] HybridClock initialized: now 1790506572646136 us; error 30 us; skew 500 ppm
Sep 27 10:56:12 dist-test-slave-tz51 krb5kdc[29964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506572, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:56:12.648860 29977 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:56:12.649637 29977 webserver.cc:460] Webserver started at http://127.14.68.62:36065/ using document root <none> and password file <none>
I20260927 10:56:12.650164 29977 fs_manager.cc:359] Metadata directory not provided
I20260927 10:56:12.650247 29977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:56:12.650413 29977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:56:12.652395 29977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "f73ce2509eb8429eb44641ab241003fd"
format_stamp: "Formatted at 2026-09-27 10:56:12 on dist-test-slave-tz51"
server_key: "b6148b77380797fbfad61a2349d2a9e3"
server_key_iv: "9a72d82c969ccc3b1600a1cb0f2032f5"
server_key_version: "encryptionkey@0"
I20260927 10:56:12.652882 29977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "f73ce2509eb8429eb44641ab241003fd"
format_stamp: "Formatted at 2026-09-27 10:56:12 on dist-test-slave-tz51"
server_key: "b6148b77380797fbfad61a2349d2a9e3"
server_key_iv: "9a72d82c969ccc3b1600a1cb0f2032f5"
server_key_version: "encryptionkey@0"
I20260927 10:56:12.656329 29977 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:56:12.658504 29992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:12.659633 29977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:56:12.659793 29977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "f73ce2509eb8429eb44641ab241003fd"
format_stamp: "Formatted at 2026-09-27 10:56:12 on dist-test-slave-tz51"
server_key: "b6148b77380797fbfad61a2349d2a9e3"
server_key_iv: "9a72d82c969ccc3b1600a1cb0f2032f5"
server_key_version: "encryptionkey@0"
I20260927 10:56:12.659880 29977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:56:12.675204 29977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:56:12.677893 29977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:56:12.678035 29977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:56:12.684949 29977 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:44105
I20260927 10:56:12.685245 30044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:44105 every 8 connection(s)
I20260927 10:56:12.685869 29977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:56:12.688508 30045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:56:12.690927 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 29977
I20260927 10:56:12.691079 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:56:12.691411 14608 external_mini_cluster.cc:1428] Setting key 9c3ea15d122dbdd1d0fc300963f883c9
I20260927 10:56:12.694422 30045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd: Bootstrap starting.
I20260927 10:56:12.697059 30045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd: Neither blocks nor log segments found. Creating new log.
I20260927 10:56:12.697851 30045 log.cc:824] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd: Log is configured to *not* fsync() on all Append() calls
I20260927 10:56:12.699535 30045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd: No bootstrap required, opened a new log
Sep 27 10:56:12 dist-test-slave-tz51 krb5kdc[29964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:56:12.703186 30045 raft_consensus.cc:348] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73ce2509eb8429eb44641ab241003fd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44105 } }
I20260927 10:56:12.703428 30045 raft_consensus.cc:374] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:56:12.703508 30045 raft_consensus.cc:729] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f73ce2509eb8429eb44641ab241003fd, State: Initialized, Role: FOLLOWER
I20260927 10:56:12.703982 30045 consensus_queue.cc:260] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73ce2509eb8429eb44641ab241003fd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44105 } }
I20260927 10:56:12.704129 30045 raft_consensus.cc:388] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:56:12.704217 30045 raft_consensus.cc:482] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:56:12.704311 30045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:56:12.705178 30045 raft_consensus.cc:504] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73ce2509eb8429eb44641ab241003fd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44105 } }
I20260927 10:56:12.705520 30045 leader_election.cc:302] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f73ce2509eb8429eb44641ab241003fd; no voters:
I20260927 10:56:12.705858 30045 leader_election.cc:288] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:56:12.705924 30050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:56:12.706141 30050 raft_consensus.cc:686] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 1 LEADER]: Becoming Leader. State: Replica: f73ce2509eb8429eb44641ab241003fd, State: Running, Role: LEADER
I20260927 10:56:12.706131 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:12.692838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40768 (local address 127.14.68.62:44105)
0927 10:56:12.693278 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:12.693297 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:12.693351 (+ 54us) server_negotiation.cc:407] Connection header received
0927 10:56:12.694135 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:12.694156 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:12.694543 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:12.694918 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:56:12.695584 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:12.697040 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:12.698059 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:12.698528 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:12.698699 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:12.701179 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:56:12.701192 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:56:12.701210 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:56:12.701236 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:56:12.702932 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:12.703367 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:12.703372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:12.703378 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:12.703554 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:12.703917 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:12.703920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:12.703922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:12.704430 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:56:12.704632 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:12.704947 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 10:56:12.705229 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":98,"threads_started":1}
I20260927 10:56:12.706458 30050 consensus_queue.cc:237] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73ce2509eb8429eb44641ab241003fd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44105 } }
I20260927 10:56:12.707070 30045 sys_catalog.cc:564] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:56:12.707538 30050 sys_catalog.cc:455] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f73ce2509eb8429eb44641ab241003fd. Latest consensus state: current_term: 1 leader_uuid: "f73ce2509eb8429eb44641ab241003fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73ce2509eb8429eb44641ab241003fd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44105 } } }
I20260927 10:56:12.707635 30050 sys_catalog.cc:458] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [sys.catalog]: This master's current role is: LEADER
I20260927 10:56:12.708494 30055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:56:12.709554 30051 sys_catalog.cc:455] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f73ce2509eb8429eb44641ab241003fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73ce2509eb8429eb44641ab241003fd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44105 } } }
I20260927 10:56:12.709673 30051 sys_catalog.cc:458] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [sys.catalog]: This master's current role is: LEADER
I20260927 10:56:12.711840 30055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:56:12.717437 30055 catalog_manager.cc:1347] Generated new cluster ID: d977a7c9f9bd4a1b8140534a5f25b8d6
I20260927 10:56:12.717519 30055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:56:12.730675 30055 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:56:12.731571 30055 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:56:12.745483 30055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd: Generated new TSK 0
I20260927 10:56:12.746169 30055 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
I20260927 10:56:12.779546 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:44105
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:41573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:56:12.892261 30071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:56:12.892511 30071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:56:12.892553 30071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:56:12.896678 30071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:56:12.896735 30071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:56:12.896822 30071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:56:12.901945 30071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:41573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:44105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:56:12.902954 30071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:56:12.904043 30071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:56:12.913641 30079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:12.913633 30077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:56:12.913866 30071 server_base.cc:1034] running on GCE node
W20260927 10:56:12.913635 30076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:56:12.914407 30071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:56:12.915006 30071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:56:12.916235 30071 hybrid_clock.cc:648] HybridClock initialized: now 1790506572916226 us; error 42 us; skew 500 ppm
Sep 27 10:56:12 dist-test-slave-tz51 krb5kdc[29964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506572, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:56:12.918915 30071 init.cc:377] Logged in from keytab as kudu/127.14.68.1@KRBTEST.COM (short username kudu)
I20260927 10:56:12.919783 30071 webserver.cc:460] Webserver started at http://127.14.68.1:45737/ using document root <none> and password file <none>
I20260927 10:56:12.920332 30071 fs_manager.cc:359] Metadata directory not provided
I20260927 10:56:12.920423 30071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:56:12.920647 30071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:56:12.922638 30071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "01a1cea0e89b4cab91b821773778f2ec"
format_stamp: "Formatted at 2026-09-27 10:56:12 on dist-test-slave-tz51"
server_key: "0f10592efc4532cdcf454986e944502b"
server_key_iv: "5472ca3d2d5bf74b6d6f81878380c73c"
server_key_version: "encryptionkey@0"
I20260927 10:56:12.923193 30071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "01a1cea0e89b4cab91b821773778f2ec"
format_stamp: "Formatted at 2026-09-27 10:56:12 on dist-test-slave-tz51"
server_key: "0f10592efc4532cdcf454986e944502b"
server_key_iv: "5472ca3d2d5bf74b6d6f81878380c73c"
server_key_version: "encryptionkey@0"
I20260927 10:56:12.926769 30071 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:56:12.929109 30086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:12.930338 30071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:56:12.930476 30071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "01a1cea0e89b4cab91b821773778f2ec"
format_stamp: "Formatted at 2026-09-27 10:56:12 on dist-test-slave-tz51"
server_key: "0f10592efc4532cdcf454986e944502b"
server_key_iv: "5472ca3d2d5bf74b6d6f81878380c73c"
server_key_version: "encryptionkey@0"
I20260927 10:56:12.930588 30071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:56:12.945806 30071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:56:12.948982 30071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:56:12.949184 30071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:56:12.949725 30071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:56:12.950501 30071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:56:12.950572 30071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:12.950636 30071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:56:12.950690 30071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:12.959275 30071 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:36069
I20260927 10:56:12.959295 30199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:36069 every 8 connection(s)
I20260927 10:56:12.960175 30071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:56:12.965606 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 30071
I20260927 10:56:12.965754 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:56:12.966084 14608 external_mini_cluster.cc:1428] Setting key 253a7304d66f18e7e56f63acc36e7a01
Sep 27 10:56:12 dist-test-slave-tz51 krb5kdc[29964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506572, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.68.2@KRBTEST.COM; defaulting to no policy
I20260927 10:56:12.976063 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:12.961884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:54247 (local address 127.14.68.62:44105)
0927 10:56:12.962042 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:12.962046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:12.964104 (+ 2058us) server_negotiation.cc:407] Connection header received
0927 10:56:12.964967 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:12.964970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:12.965017 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:12.965129 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:56:12.966656 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:12.967195 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:12.968883 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:12.969132 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:12.969285 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:12.972143 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:56:12.972146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:56:12.972148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:56:12.972170 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:56:12.973602 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:12.974227 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:12.974229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:12.974231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:12.974306 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:12.974706 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:12.974708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:12.974709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:12.974956 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:56:12.975031 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:12.975777 (+ 746us) server_negotiation.cc:299] Negotiation successful
0927 10:56:12.975883 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 10:56:12.976831 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:12.962241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.1:54247)
0927 10:56:12.963941 (+ 1700us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:12.963982 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:12.964776 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:12.965294 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:12.965300 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:12.965820 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:56:12.966502 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:12.966524 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:12.967326 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:12.967330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:12.968740 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:12.968755 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:12.969317 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:12.969321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:12.969495 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:12.970097 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:56:12.970115 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:56:12.971941 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:56:12.973722 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:12.973732 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:12.973747 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:12.974112 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:12.974399 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:12.974402 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:12.974405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:12.974605 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:12.975072 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:56:12.975080 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:56:12.975433 (+ 353us) client_negotiation.cc:769] Sending connection context
0927 10:56:12.975642 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 10:56:12.976023 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1512,"thread_start_us":693,"threads_started":1}
I20260927 10:56:12.978121 30200 heartbeater.cc:347] Connected to a master server at 127.14.68.62:44105
I20260927 10:56:12.978325 30200 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.2@KRBTEST.COM" created.
I20260927 10:56:12.978924 30200 heartbeater.cc:511] Master 127.14.68.62:44105 requested a full tablet report, sending...
I20260927 10:56:12.980574 30009 ts_manager.cc:194] Registered new tserver with Master: 01a1cea0e89b4cab91b821773778f2ec (127.14.68.1:36069)
I20260927 10:56:12.981712 30009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:54247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
I20260927 10:56:12.993142 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:44105
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:41573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:56:13.100152 30205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:56:13.100422 30205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:56:13.100497 30205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:56:13.104388 30205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:56:13.104470 30205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:56:13.104555 30205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:56:13.109812 30205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:41573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:44105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:56:13.110805 30205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:56:13.111790 30205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:56:13.120994 30213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:13.121017 30210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:13.121008 30211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:56:13.121065 30205 server_base.cc:1034] running on GCE node
I20260927 10:56:13.121610 30205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:56:13.122219 30205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:56:13.123409 30205 hybrid_clock.cc:648] HybridClock initialized: now 1790506573123397 us; error 34 us; skew 500 ppm
Sep 27 10:56:13 dist-test-slave-tz51 krb5kdc[29964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506573, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:56:13.126173 30205 init.cc:377] Logged in from keytab as kudu/127.14.68.2@KRBTEST.COM (short username kudu)
I20260927 10:56:13.126982 30205 webserver.cc:460] Webserver started at http://127.14.68.2:42189/ using document root <none> and password file <none>
I20260927 10:56:13.127534 30205 fs_manager.cc:359] Metadata directory not provided
I20260927 10:56:13.127624 30205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:56:13.127858 30205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:56:13.129801 30205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "f2712cbe30ef4aee8e52e90b65ecee5e"
format_stamp: "Formatted at 2026-09-27 10:56:13 on dist-test-slave-tz51"
server_key: "2e0f89edc428fb4575dcb5a09a905bc4"
server_key_iv: "99c327b4650d60c1ee0bcfd5f4961393"
server_key_version: "encryptionkey@0"
I20260927 10:56:13.130352 30205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "f2712cbe30ef4aee8e52e90b65ecee5e"
format_stamp: "Formatted at 2026-09-27 10:56:13 on dist-test-slave-tz51"
server_key: "2e0f89edc428fb4575dcb5a09a905bc4"
server_key_iv: "99c327b4650d60c1ee0bcfd5f4961393"
server_key_version: "encryptionkey@0"
I20260927 10:56:13.133881 30205 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:56:13.136551 30220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:13.137692 30205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:56:13.137832 30205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "f2712cbe30ef4aee8e52e90b65ecee5e"
format_stamp: "Formatted at 2026-09-27 10:56:13 on dist-test-slave-tz51"
server_key: "2e0f89edc428fb4575dcb5a09a905bc4"
server_key_iv: "99c327b4650d60c1ee0bcfd5f4961393"
server_key_version: "encryptionkey@0"
I20260927 10:56:13.137944 30205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:56:13.148550 30205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:56:13.151829 30205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:56:13.152019 30205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:56:13.152603 30205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:56:13.153402 30205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:56:13.153481 30205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:13.153544 30205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:56:13.153596 30205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:13.162379 30205 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:43669
I20260927 10:56:13.162454 30333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:43669 every 8 connection(s)
I20260927 10:56:13.163306 30205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:56:13.168896 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 30205
I20260927 10:56:13.169031 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:56:13.169386 14608 external_mini_cluster.cc:1428] Setting key 0425a3c7ee02d16f5ff69f8ab0ba71ee
Sep 27 10:56:13 dist-test-slave-tz51 krb5kdc[29964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506573, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:56:13.177295 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:13.165144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:39131 (local address 127.14.68.62:44105)
0927 10:56:13.165281 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:13.165285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:13.166271 (+ 986us) server_negotiation.cc:407] Connection header received
0927 10:56:13.167138 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:13.167142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:13.167198 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:13.167302 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:56:13.168446 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.169219 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.170630 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.170936 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.171046 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:13.173439 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:56:13.173442 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:56:13.173445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:56:13.173467 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:56:13.175040 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:13.175627 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:13.175630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:13.175631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:13.175749 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:13.176170 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:13.176175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:13.176177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:13.176374 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:56:13.176483 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:13.176973 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 10:56:13.177127 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:56:13.178050 30336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:13.165476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.2:39131)
0927 10:56:13.166117 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:13.166155 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:13.166938 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:13.167466 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:13.167473 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:13.167839 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:56:13.168299 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:13.168312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.169353 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.169356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:13.170501 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:13.170511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.171087 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.171090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:13.171230 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:13.171825 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:56:13.171841 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:56:13.173266 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:56:13.175158 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:13.175164 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:13.175179 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:13.175499 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:13.175853 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:13.175856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:13.175858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:13.176067 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:13.176481 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:56:13.176487 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:56:13.176738 (+ 251us) client_negotiation.cc:769] Sending connection context
0927 10:56:13.176929 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 10:56:13.177246 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.14.68.3@KRBTEST.COM; defaulting to no policy
I20260927 10:56:13.179240 30334 heartbeater.cc:347] Connected to a master server at 127.14.68.62:44105
I20260927 10:56:13.179448 30334 heartbeater.cc:464] Registering TS with master...
I20260927 10:56:13.180037 30334 heartbeater.cc:511] Master 127.14.68.62:44105 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.3@KRBTEST.COM" created.
I20260927 10:56:13.181232 30009 ts_manager.cc:194] Registered new tserver with Master: f2712cbe30ef4aee8e52e90b65ecee5e (127.14.68.2:43669)
I20260927 10:56:13.181845 30009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:39131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
I20260927 10:56:13.198527 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:44105
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:41573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:56:13.304018 30339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:56:13.304245 30339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:56:13.304286 30339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:56:13.308450 30339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:56:13.308499 30339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:56:13.308589 30339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:56:13.313840 30339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:41573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:44105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:56:13.314826 30339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:56:13.315860 30339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:56:13.325419 30347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:13.325456 30344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:13.325439 30345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:56:13.325670 30339 server_base.cc:1034] running on GCE node
I20260927 10:56:13.326196 30339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:56:13.326821 30339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:56:13.327998 30339 hybrid_clock.cc:648] HybridClock initialized: now 1790506573327975 us; error 47 us; skew 500 ppm
Sep 27 10:56:13 dist-test-slave-tz51 krb5kdc[29964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506573, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:56:13.330724 30339 init.cc:377] Logged in from keytab as kudu/127.14.68.3@KRBTEST.COM (short username kudu)
I20260927 10:56:13.331555 30339 webserver.cc:460] Webserver started at http://127.14.68.3:35035/ using document root <none> and password file <none>
I20260927 10:56:13.332186 30339 fs_manager.cc:359] Metadata directory not provided
I20260927 10:56:13.332278 30339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:56:13.332504 30339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:56:13.334640 30339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "e6690d948ed24e9bb78f0ca93fca22c2"
format_stamp: "Formatted at 2026-09-27 10:56:13 on dist-test-slave-tz51"
server_key: "a25abb69c6ae4e67b62638274a11cbf1"
server_key_iv: "0cdf16de2ac65c9c72faac68d9641206"
server_key_version: "encryptionkey@0"
I20260927 10:56:13.335250 30339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "e6690d948ed24e9bb78f0ca93fca22c2"
format_stamp: "Formatted at 2026-09-27 10:56:13 on dist-test-slave-tz51"
server_key: "a25abb69c6ae4e67b62638274a11cbf1"
server_key_iv: "0cdf16de2ac65c9c72faac68d9641206"
server_key_version: "encryptionkey@0"
I20260927 10:56:13.338841 30339 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:56:13.341274 30354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:13.342620 30339 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:56:13.342767 30339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "e6690d948ed24e9bb78f0ca93fca22c2"
format_stamp: "Formatted at 2026-09-27 10:56:13 on dist-test-slave-tz51"
server_key: "a25abb69c6ae4e67b62638274a11cbf1"
server_key_iv: "0cdf16de2ac65c9c72faac68d9641206"
server_key_version: "encryptionkey@0"
I20260927 10:56:13.342886 30339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:56:13.358626 30339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:56:13.361990 30339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:56:13.362200 30339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:56:13.362843 30339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:56:13.363646 30339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:56:13.363752 30339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:13.363848 30339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:56:13.363899 30339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:13.372229 30339 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:35715
I20260927 10:56:13.372299 30467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:35715 every 8 connection(s)
I20260927 10:56:13.373137 30339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:56:13.374225 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 30339
I20260927 10:56:13.374346 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:56:13.374625 14608 external_mini_cluster.cc:1428] Setting key 88709143ec84644d9c0c120d603be1db
Sep 27 10:56:13 dist-test-slave-tz51 krb5kdc[29964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506573, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:56:13.385731 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:13.374976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:34463 (local address 127.14.68.62:44105)
0927 10:56:13.375124 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:13.375128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:13.376309 (+ 1181us) server_negotiation.cc:407] Connection header received
0927 10:56:13.377152 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:13.377155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:13.377200 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:13.377265 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:56:13.378365 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.378903 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.379984 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.380205 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.380286 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:13.382460 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:56:13.382463 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:56:13.382465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:56:13.382487 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:56:13.384068 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:13.384516 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:13.384519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:13.384520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:13.384600 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:13.384881 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:13.384883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:13.384884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:13.385039 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:56:13.385122 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:13.385524 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 10:56:13.385622 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:56:13.386461 30470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:13.375385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.3:34463)
0927 10:56:13.376159 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:13.376209 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:13.376980 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:13.377393 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:13.377399 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:13.377740 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:56:13.378220 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:13.378234 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.379008 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.379010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:13.379885 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:13.379894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.380327 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.380329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:13.380460 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:13.380944 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:56:13.380958 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:56:13.382319 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:56:13.384164 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:13.384168 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:13.384178 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:13.384416 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:13.384677 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:13.384679 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:13.384681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:13.384803 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:13.385129 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:56:13.385133 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:56:13.385337 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 10:56:13.385479 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 10:56:13.385694 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":146,"threads_started":1}
I20260927 10:56:13.387594 30468 heartbeater.cc:347] Connected to a master server at 127.14.68.62:44105
I20260927 10:56:13.387861 30468 heartbeater.cc:464] Registering TS with master...
I20260927 10:56:13.388403 30468 heartbeater.cc:511] Master 127.14.68.62:44105 requested a full tablet report, sending...
I20260927 10:56:13.389479 30009 ts_manager.cc:194] Registered new tserver with Master: e6690d948ed24e9bb78f0ca93fca22c2 (127.14.68.3:35715)
I20260927 10:56:13.390015 30009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:34463
I20260927 10:56:13.400060 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:56:13.409107 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:13.401895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40770 (local address 127.14.68.62:44105)
0927 10:56:13.402068 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:13.402072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:13.402183 (+ 111us) server_negotiation.cc:407] Connection header received
0927 10:56:13.402245 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:13.402248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:13.402304 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:13.402370 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:56:13.402988 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.403487 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.404303 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.404511 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.404596 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:13.405806 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:56:13.405809 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:56:13.405811 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:56:13.405832 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:56:13.407329 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:13.407810 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:13.407813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:13.407814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:13.407888 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:13.408190 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:13.408192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:13.408193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:13.408438 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:56:13.408597 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:13.408837 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 10:56:13.408948 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 10:56:13.411571 30009 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:56:13.414520 30009 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:56:13.429188 30479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:13.422587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:36069 (local address 127.0.0.1:42914)
0927 10:56:13.423499 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:13.423547 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:13.423668 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:13.424560 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:13.424566 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:13.424588 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:56:13.424748 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:13.424755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.426245 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.426248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:13.427580 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:13.427589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.428660 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.428663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:13.428790 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:13.428832 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 10:56:13.428939 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 10:56:13.429045 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":3,"thread_start_us":78,"threads_started":1}
I20260927 10:56:13.429225 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:13.423364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42914 (local address 127.14.68.1:36069)
0927 10:56:13.424068 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:13.424080 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:13.424104 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:56:13.424178 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:13.424185 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:13.424395 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:13.424555 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:56:13.424887 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.426108 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.427719 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.428529 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.428660 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:13.428814 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:13.428985 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 10:56:13.429075 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":91,"threads_started":1}
I20260927 10:56:13.429903 30482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:13.422658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:35715 (local address 127.0.0.1:34574)
0927 10:56:13.424452 (+ 1794us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:13.424463 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:13.424560 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:13.425884 (+ 1324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:13.425886 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:13.425903 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:56:13.425991 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:13.425997 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.427414 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.427417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:13.428304 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:13.428313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.429513 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.429516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:13.429609 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:13.429629 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:56:13.429716 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 10:56:13.429781 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1745,"mutex_wait_us":22,"thread_start_us":107,"threads_started":1}
I20260927 10:56:13.429958 30484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:13.423016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34574 (local address 127.14.68.3:35715)
0927 10:56:13.425490 (+ 2474us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:13.425504 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:13.425527 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:56:13.425607 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:13.425611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:13.425756 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:13.425911 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:56:13.426100 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.427266 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.428437 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.429414 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.429519 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:13.429654 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:13.429741 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 10:56:13.429826 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2345,"thread_start_us":2038,"threads_started":1}
I20260927 10:56:13.430665 30480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:13.423646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:43669 (local address 127.0.0.1:36804)
0927 10:56:13.425342 (+ 1696us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:13.425356 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:13.425468 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:13.426050 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:13.426053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:13.426069 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:56:13.426169 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:13.426178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.427337 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.427342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:13.428570 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:13.428582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.430339 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.430343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:13.430422 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:13.430443 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:56:13.430520 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 10:56:13.430579 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1628,"mutex_wait_us":52,"thread_start_us":116,"threads_started":1}
I20260927 10:56:13.430766 30483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:13.424075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36804 (local address 127.14.68.2:43669)
0927 10:56:13.424828 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:13.424842 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:13.425466 (+ 624us) server_negotiation.cc:407] Connection header received
0927 10:56:13.425573 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:13.425578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:13.425759 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:13.425893 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:56:13.426297 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.427212 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.428713 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.430208 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.430315 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:13.430451 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:13.430546 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 10:56:13.430637 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":163,"threads_started":1}
I20260927 10:56:13.432335 30134 tablet_service.cc:1467] Processing CreateTablet for tablet 37e79781aec5463880e711a8a0563645 (DEFAULT_TABLE table=test-table [id=f4ea637431e744f897d0a57bde5e779b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:56:13.433246 30134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e79781aec5463880e711a8a0563645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:56:13.435998 30268 tablet_service.cc:1467] Processing CreateTablet for tablet 37e79781aec5463880e711a8a0563645 (DEFAULT_TABLE table=test-table [id=f4ea637431e744f897d0a57bde5e779b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:56:13.435935 30402 tablet_service.cc:1467] Processing CreateTablet for tablet 37e79781aec5463880e711a8a0563645 (DEFAULT_TABLE table=test-table [id=f4ea637431e744f897d0a57bde5e779b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:56:13.436817 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e79781aec5463880e711a8a0563645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:56:13.436937 30402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e79781aec5463880e711a8a0563645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:56:13.440445 30485 tablet_bootstrap.cc:492] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec: Bootstrap starting.
I20260927 10:56:13.442910 30485 tablet_bootstrap.cc:654] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec: Neither blocks nor log segments found. Creating new log.
I20260927 10:56:13.443204 30486 tablet_bootstrap.cc:492] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e: Bootstrap starting.
I20260927 10:56:13.444305 30487 tablet_bootstrap.cc:492] T 37e79781aec5463880e711a8a0563645 P e6690d948ed24e9bb78f0ca93fca22c2: Bootstrap starting.
I20260927 10:56:13.444682 30485 log.cc:824] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec: Log is configured to *not* fsync() on all Append() calls
I20260927 10:56:13.445762 30486 tablet_bootstrap.cc:654] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e: Neither blocks nor log segments found. Creating new log.
I20260927 10:56:13.446213 30487 tablet_bootstrap.cc:654] T 37e79781aec5463880e711a8a0563645 P e6690d948ed24e9bb78f0ca93fca22c2: Neither blocks nor log segments found. Creating new log.
I20260927 10:56:13.446533 30486 log.cc:824] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e: Log is configured to *not* fsync() on all Append() calls
I20260927 10:56:13.446873 30485 tablet_bootstrap.cc:492] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec: No bootstrap required, opened a new log
I20260927 10:56:13.447096 30485 ts_tablet_manager.cc:1397] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 10:56:13.447363 30487 log.cc:824] T 37e79781aec5463880e711a8a0563645 P e6690d948ed24e9bb78f0ca93fca22c2: Log is configured to *not* fsync() on all Append() calls
I20260927 10:56:13.448400 30486 tablet_bootstrap.cc:492] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e: No bootstrap required, opened a new log
I20260927 10:56:13.448589 30486 ts_tablet_manager.cc:1397] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:56:13.449232 30487 tablet_bootstrap.cc:492] T 37e79781aec5463880e711a8a0563645 P e6690d948ed24e9bb78f0ca93fca22c2: No bootstrap required, opened a new log
I20260927 10:56:13.449399 30487 ts_tablet_manager.cc:1397] T 37e79781aec5463880e711a8a0563645 P e6690d948ed24e9bb78f0ca93fca22c2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:56:13.451116 30485 raft_consensus.cc:348] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } }
I20260927 10:56:13.451392 30485 raft_consensus.cc:374] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:56:13.451478 30485 raft_consensus.cc:729] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a1cea0e89b4cab91b821773778f2ec, State: Initialized, Role: FOLLOWER
I20260927 10:56:13.451874 30486 raft_consensus.cc:348] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } }
I20260927 10:56:13.451968 30485 consensus_queue.cc:260] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } }
I20260927 10:56:13.452086 30486 raft_consensus.cc:374] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:56:13.452164 30486 raft_consensus.cc:729] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2712cbe30ef4aee8e52e90b65ecee5e, State: Initialized, Role: FOLLOWER
I20260927 10:56:13.452544 30486 consensus_queue.cc:260] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } }
I20260927 10:56:13.452925 30485 ts_tablet_manager.cc:1428] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:56:13.453121 30334 heartbeater.cc:503] Master 127.14.68.62:44105 was elected leader, sending a full tablet report...
I20260927 10:56:13.453032 30487 raft_consensus.cc:348] T 37e79781aec5463880e711a8a0563645 P e6690d948ed24e9bb78f0ca93fca22c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } }
I20260927 10:56:13.453250 30487 raft_consensus.cc:374] T 37e79781aec5463880e711a8a0563645 P e6690d948ed24e9bb78f0ca93fca22c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:56:13.453312 30200 heartbeater.cc:503] Master 127.14.68.62:44105 was elected leader, sending a full tablet report...
I20260927 10:56:13.453318 30487 raft_consensus.cc:729] T 37e79781aec5463880e711a8a0563645 P e6690d948ed24e9bb78f0ca93fca22c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6690d948ed24e9bb78f0ca93fca22c2, State: Initialized, Role: FOLLOWER
I20260927 10:56:13.453424 30486 ts_tablet_manager.cc:1428] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:56:13.453730 30487 consensus_queue.cc:260] T 37e79781aec5463880e711a8a0563645 P e6690d948ed24e9bb78f0ca93fca22c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } }
I20260927 10:56:13.454550 30487 ts_tablet_manager.cc:1428] T 37e79781aec5463880e711a8a0563645 P e6690d948ed24e9bb78f0ca93fca22c2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:56:13.454551 30468 heartbeater.cc:503] Master 127.14.68.62:44105 was elected leader, sending a full tablet report...
W20260927 10:56:13.461221 30201 tablet.cc:2369] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:56:13.466325 30492 raft_consensus.cc:482] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:56:13.466481 30492 raft_consensus.cc:504] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } }
I20260927 10:56:13.467414 30492 leader_election.cc:288] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01a1cea0e89b4cab91b821773778f2ec (127.14.68.1:36069), e6690d948ed24e9bb78f0ca93fca22c2 (127.14.68.3:35715)
I20260927 10:56:13.472366 30484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:13.467816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:55691 (local address 127.14.68.3:35715)
0927 10:56:13.467922 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:13.467926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:13.468209 (+ 283us) server_negotiation.cc:407] Connection header received
0927 10:56:13.468325 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:13.468327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:13.468365 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:13.468447 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:56:13.468663 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.469365 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.471066 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.471672 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.471804 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:13.471865 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:13.472172 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 10:56:13.472245 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 10:56:13.472366 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:13.467850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:57509 (local address 127.14.68.1:36069)
0927 10:56:13.467968 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:13.467972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:13.467991 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:56:13.468047 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:13.468049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:13.468086 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:13.468151 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:56:13.468461 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.469262 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.471066 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.471672 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.471804 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:13.471865 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:13.472177 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 10:56:13.472248 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:56:13.472411 30336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:13.467635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:36069 (local address 127.14.68.2:57509)
0927 10:56:13.467844 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:13.467860 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:13.467946 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:13.468196 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:13.468198 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:13.468228 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:56:13.468361 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:13.468367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.469356 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.469359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:13.470941 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:13.470952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.471799 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.471803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:13.471991 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:13.472063 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 10:56:13.472143 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 10:56:13.472258 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":226,"spinlock_wait_cycles":1408}
I20260927 10:56:13.472420 30494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:13.467639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:35715 (local address 127.14.68.2:55691)
0927 10:56:13.468095 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:13.468105 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:13.468234 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:13.468459 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:13.468464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:13.468480 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:56:13.468566 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:13.468570 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.469456 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.469459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:13.470941 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:13.470952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.471799 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.471803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:13.471971 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:13.472067 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 10:56:13.472143 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 10:56:13.472258 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":378,"spinlock_wait_cycles":29312,"thread_start_us":105,"threads_started":1}
I20260927 10:56:13.473001 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e79781aec5463880e711a8a0563645" candidate_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" is_pre_election: true
I20260927 10:56:13.472999 30154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e79781aec5463880e711a8a0563645" candidate_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01a1cea0e89b4cab91b821773778f2ec" is_pre_election: true
I20260927 10:56:13.473259 30422 raft_consensus.cc:2454] T 37e79781aec5463880e711a8a0563645 P e6690d948ed24e9bb78f0ca93fca22c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2712cbe30ef4aee8e52e90b65ecee5e in term 0.
I20260927 10:56:13.473290 30154 raft_consensus.cc:2454] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2712cbe30ef4aee8e52e90b65ecee5e in term 0.
I20260927 10:56:13.473701 30221 leader_election.cc:302] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6690d948ed24e9bb78f0ca93fca22c2, f2712cbe30ef4aee8e52e90b65ecee5e; no voters:
I20260927 10:56:13.473953 30492 raft_consensus.cc:2781] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:56:13.474033 30492 raft_consensus.cc:482] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:56:13.474076 30492 raft_consensus.cc:3037] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:56:13.475085 30492 raft_consensus.cc:504] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } }
I20260927 10:56:13.475451 30492 leader_election.cc:288] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e [CANDIDATE]: Term 1 election: Requested vote from peers 01a1cea0e89b4cab91b821773778f2ec (127.14.68.1:36069), e6690d948ed24e9bb78f0ca93fca22c2 (127.14.68.3:35715)
I20260927 10:56:13.475847 30154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e79781aec5463880e711a8a0563645" candidate_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01a1cea0e89b4cab91b821773778f2ec"
I20260927 10:56:13.475833 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e79781aec5463880e711a8a0563645" candidate_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6690d948ed24e9bb78f0ca93fca22c2"
I20260927 10:56:13.475961 30154 raft_consensus.cc:3037] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:56:13.475961 30422 raft_consensus.cc:3037] T 37e79781aec5463880e711a8a0563645 P e6690d948ed24e9bb78f0ca93fca22c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:56:13.477276 30154 raft_consensus.cc:2454] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2712cbe30ef4aee8e52e90b65ecee5e in term 1.
I20260927 10:56:13.477272 30422 raft_consensus.cc:2454] T 37e79781aec5463880e711a8a0563645 P e6690d948ed24e9bb78f0ca93fca22c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2712cbe30ef4aee8e52e90b65ecee5e in term 1.
I20260927 10:56:13.477581 30224 leader_election.cc:302] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a1cea0e89b4cab91b821773778f2ec, f2712cbe30ef4aee8e52e90b65ecee5e; no voters:
I20260927 10:56:13.477811 30492 raft_consensus.cc:2781] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:56:13.478013 30492 raft_consensus.cc:686] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e [term 1 LEADER]: Becoming Leader. State: Replica: f2712cbe30ef4aee8e52e90b65ecee5e, State: Running, Role: LEADER
I20260927 10:56:13.478349 30492 consensus_queue.cc:237] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } }
I20260927 10:56:13.481541 30008 catalog_manager.cc:5495] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e reported cstate change: term changed from 0 to 1, leader changed from <none> to f2712cbe30ef4aee8e52e90b65ecee5e (127.14.68.2). New cstate: current_term: 1 leader_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } health_report { overall_health: HEALTHY } } }
I20260927 10:56:13.505149 30483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:13.502040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.14.68.2:43669)
0927 10:56:13.502169 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:13.502172 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:13.502185 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:56:13.502255 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:13.502257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:13.502307 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:13.502401 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:56:13.502600 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.503078 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.503895 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:13.504122 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:13.504197 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:13.504436 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:56:13.504834 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:56:13.504921 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:13.504973 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 10:56:13.505019 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:56:13.511003 30154 raft_consensus.cc:1261] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec [term 1 FOLLOWER]: Refusing update from remote peer f2712cbe30ef4aee8e52e90b65ecee5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:56:13.511003 30422 raft_consensus.cc:1261] T 37e79781aec5463880e711a8a0563645 P e6690d948ed24e9bb78f0ca93fca22c2 [term 1 FOLLOWER]: Refusing update from remote peer f2712cbe30ef4aee8e52e90b65ecee5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:56:13.511687 30492 consensus_queue.cc:1035] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:56:13.511916 30495 consensus_queue.cc:1035] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:56:13.518955 30498 mvcc.cc:205] Tried to move back new op lower bound from 7333914925094359040 to 7333914924969590784. Current Snapshot: MvccSnapshot[applied={T|T < 7333914925094359040}]
I20260927 10:56:13.519737 30499 mvcc.cc:205] Tried to move back new op lower bound from 7333914925094359040 to 7333914924969590784. Current Snapshot: MvccSnapshot[applied={T|T < 7333914925094359040}]
I20260927 10:56:13.521891 30500 mvcc.cc:205] Tried to move back new op lower bound from 7333914925094359040 to 7333914924969590784. Current Snapshot: MvccSnapshot[applied={T|T < 7333914925094359040}]
I20260927 10:56:13.527948 30007 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:56:13.528121 30007 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:56:13.530390 30007 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd: Sending DeleteTablet for 3 replicas of tablet 37e79781aec5463880e711a8a0563645
I20260927 10:56:13.531136 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 30071
I20260927 10:56:13.531098 30402 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e79781aec5463880e711a8a0563645 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:56:13 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:34574
I20260927 10:56:13.531136 30134 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e79781aec5463880e711a8a0563645 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:56:13 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:42914
I20260927 10:56:13.531876 30508 tablet_replica.cc:330] T 37e79781aec5463880e711a8a0563645 P e6690d948ed24e9bb78f0ca93fca22c2: stopping tablet replica
I20260927 10:56:13.532166 30508 raft_consensus.cc:2229] T 37e79781aec5463880e711a8a0563645 P e6690d948ed24e9bb78f0ca93fca22c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:56:13.532584 30508 raft_consensus.cc:2258] T 37e79781aec5463880e711a8a0563645 P e6690d948ed24e9bb78f0ca93fca22c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:56:13.532938 30268 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e79781aec5463880e711a8a0563645 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:56:13 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:36804
I20260927 10:56:13.533370 30509 tablet_replica.cc:330] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e: stopping tablet replica
I20260927 10:56:13.533692 30509 raft_consensus.cc:2229] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:56:13.534073 30509 raft_consensus.cc:2258] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:56:13.534613 30508 ts_tablet_manager.cc:1905] T 37e79781aec5463880e711a8a0563645 P e6690d948ed24e9bb78f0ca93fca22c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:56:13.535470 30509 ts_tablet_manager.cc:1905] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:56:13.538096 30508 ts_tablet_manager.cc:1918] T 37e79781aec5463880e711a8a0563645 P e6690d948ed24e9bb78f0ca93fca22c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:56:13.538121 30509 ts_tablet_manager.cc:1918] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:56:13.538200 30509 log.cc:1196] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/37e79781aec5463880e711a8a0563645
I20260927 10:56:13.538280 30508 log.cc:1196] T 37e79781aec5463880e711a8a0563645 P e6690d948ed24e9bb78f0ca93fca22c2: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/37e79781aec5463880e711a8a0563645
I20260927 10:56:13.538618 30509 ts_tablet_manager.cc:1939] T 37e79781aec5463880e711a8a0563645 P f2712cbe30ef4aee8e52e90b65ecee5e: Deleting consensus metadata
I20260927 10:56:13.538704 30508 ts_tablet_manager.cc:1939] T 37e79781aec5463880e711a8a0563645 P e6690d948ed24e9bb78f0ca93fca22c2: Deleting consensus metadata
W20260927 10:56:13.539181 29996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.68.1:36069 (error 108)
I20260927 10:56:13.539422 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 30205
I20260927 10:56:13.540781 30480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:13.540482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:36069 (local address 127.0.0.1:42924)
0927 10:56:13.540603 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:13.540685 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.1:36069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 10:56:13.540962 29996 catalog_manager.cc:4584] TS 01a1cea0e89b4cab91b821773778f2ec (127.14.68.1:36069): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 37e79781aec5463880e711a8a0563645: Network error: Client connection negotiation failed: client connection to 127.14.68.1:36069: connect: Connection refused (error 111)
I20260927 10:56:13.541429 29993 catalog_manager.cc:4841] TS e6690d948ed24e9bb78f0ca93fca22c2 (127.14.68.3:35715): tablet 37e79781aec5463880e711a8a0563645 (table test-table [id=f4ea637431e744f897d0a57bde5e779b]) successfully deleted
I20260927 10:56:13.544716 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 30339
I20260927 10:56:13.545825 30480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:13.544892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:43669 (local address 127.0.0.1:36824)
0927 10:56:13.545668 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:13.545732 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.2:43669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":683}
I20260927 10:56:13.550343 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 29977
I20260927 10:56:33.556819 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:44105
--webserver_interface=127.14.68.62
--webserver_port=36065
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:41573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:44105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:56:33.664790 30511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:56:33.665060 30511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:56:33.665102 30511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:56:33.669114 30511 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 10:56:33.669162 30511 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 10:56:33.669178 30511 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 10:56:33.669191 30511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:56:33.669209 30511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:56:33.669282 30511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:56:33.669310 30511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:56:33.669343 30511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:56:33.674728 30511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:41573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:44105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:44105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=36065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:56:33.675840 30511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:56:33.676918 30511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:56:33.690436 30518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:33.690505 30520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:56:33.690511 30511 server_base.cc:1034] running on GCE node
W20260927 10:56:33.690449 30517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:56:33.691438 30511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:56:33.692662 30511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:56:33.693828 30511 hybrid_clock.cc:648] HybridClock initialized: now 1790506593693809 us; error 30 us; skew 500 ppm
Sep 27 10:56:33 dist-test-slave-tz51 krb5kdc[29964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506593, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:56:33.696689 30511 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:56:33.697649 30511 webserver.cc:460] Webserver started at http://127.14.68.62:36065/ using document root <none> and password file <none>
I20260927 10:56:33.698253 30511 fs_manager.cc:359] Metadata directory not provided
I20260927 10:56:33.698367 30511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:56:33.702427 30511 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:56:33.704845 30527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:33.706311 30511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:56:33.706473 30511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "f73ce2509eb8429eb44641ab241003fd"
format_stamp: "Formatted at 2026-09-27 10:56:12 on dist-test-slave-tz51"
server_key: "b6148b77380797fbfad61a2349d2a9e3"
server_key_iv: "9a72d82c969ccc3b1600a1cb0f2032f5"
server_key_version: "encryptionkey@0"
I20260927 10:56:33.707096 30511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:56:33.723477 30511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:56:33.726574 30511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:56:33.726805 30511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:56:33.734534 30511 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:44105
I20260927 10:56:33.734885 30579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:44105 every 8 connection(s)
I20260927 10:56:33.735766 30511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:56:33.740579 30580 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:56:33.742200 30580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd: Bootstrap starting.
I20260927 10:56:33.743534 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 30511
I20260927 10:56:33.744062 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:36069
--local_ip_for_outbound_sockets=127.14.68.1
--tserver_master_addrs=127.14.68.62:44105
--webserver_port=45737
--webserver_interface=127.14.68.1
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:41573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:56:33.760025 30580 log.cc:824] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd: Log is configured to *not* fsync() on all Append() calls
I20260927 10:56:33.768903 30580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:56:33.769291 30580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd: Bootstrap complete.
I20260927 10:56:33.773303 30580 raft_consensus.cc:348] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73ce2509eb8429eb44641ab241003fd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44105 } }
I20260927 10:56:33.773681 30580 raft_consensus.cc:729] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f73ce2509eb8429eb44641ab241003fd, State: Initialized, Role: FOLLOWER
I20260927 10:56:33.774210 30580 consensus_queue.cc:260] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73ce2509eb8429eb44641ab241003fd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44105 } }
I20260927 10:56:33.774327 30580 raft_consensus.cc:388] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:56:33.774420 30580 raft_consensus.cc:482] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:56:33.774535 30580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:56:33.776515 30580 raft_consensus.cc:504] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73ce2509eb8429eb44641ab241003fd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44105 } }
I20260927 10:56:33.776842 30580 leader_election.cc:302] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f73ce2509eb8429eb44641ab241003fd; no voters:
I20260927 10:56:33.777179 30580 leader_election.cc:288] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 10:56:33.777257 30584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 2 FOLLOWER]: Leader election won for term 2
I20260927 10:56:33.777556 30584 raft_consensus.cc:686] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 2 LEADER]: Becoming Leader. State: Replica: f73ce2509eb8429eb44641ab241003fd, State: Running, Role: LEADER
I20260927 10:56:33.777905 30584 consensus_queue.cc:237] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73ce2509eb8429eb44641ab241003fd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44105 } }
I20260927 10:56:33.778384 30580 sys_catalog.cc:564] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:56:33.779333 30586 sys_catalog.cc:455] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f73ce2509eb8429eb44641ab241003fd. Latest consensus state: current_term: 2 leader_uuid: "f73ce2509eb8429eb44641ab241003fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73ce2509eb8429eb44641ab241003fd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44105 } } }
I20260927 10:56:33.779451 30586 sys_catalog.cc:458] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [sys.catalog]: This master's current role is: LEADER
I20260927 10:56:33.779727 30585 sys_catalog.cc:455] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f73ce2509eb8429eb44641ab241003fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73ce2509eb8429eb44641ab241003fd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44105 } } }
I20260927 10:56:33.779808 30585 sys_catalog.cc:458] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [sys.catalog]: This master's current role is: LEADER
I20260927 10:56:33.779834 30593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:56:33.784574 30593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:56:33.785276 30593 catalog_manager.cc:1259] Loaded cluster ID: d977a7c9f9bd4a1b8140534a5f25b8d6
I20260927 10:56:33.785346 30593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:56:33.787853 30593 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:56:33.788560 30593 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd: Loaded TSK: 0
I20260927 10:56:33.806325 30593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd: Generated new TSK 1
I20260927 10:56:33.806602 30593 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:56:33.869463 30582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:56:33.869782 30582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:56:33.869891 30582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:56:33.873960 30582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:56:33.874058 30582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:56:33.874150 30582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:56:33.879485 30582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:41573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:36069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=45737
--tserver_master_addrs=127.14.68.62:44105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:56:33.880681 30582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:56:33.881776 30582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:56:33.891348 30610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:33.891357 30607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:33.891366 30608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:56:33.891619 30582 server_base.cc:1034] running on GCE node
I20260927 10:56:33.892074 30582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:56:33.892763 30582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:56:33.893989 30582 hybrid_clock.cc:648] HybridClock initialized: now 1790506593893957 us; error 43 us; skew 500 ppm
Sep 27 10:56:33 dist-test-slave-tz51 krb5kdc[29964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506593, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:56:33.897269 30582 init.cc:377] Logged in from keytab as kudu/127.14.68.1@KRBTEST.COM (short username kudu)
I20260927 10:56:33.898205 30582 webserver.cc:460] Webserver started at http://127.14.68.1:45737/ using document root <none> and password file <none>
I20260927 10:56:33.898835 30582 fs_manager.cc:359] Metadata directory not provided
I20260927 10:56:33.898958 30582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:56:33.903867 30582 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:56:33.906464 30617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:33.908030 30582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:56:33.908177 30582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "01a1cea0e89b4cab91b821773778f2ec"
format_stamp: "Formatted at 2026-09-27 10:56:12 on dist-test-slave-tz51"
server_key: "0f10592efc4532cdcf454986e944502b"
server_key_iv: "5472ca3d2d5bf74b6d6f81878380c73c"
server_key_version: "encryptionkey@0"
I20260927 10:56:33.908800 30582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:56:33.930114 30582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:56:33.933826 30582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:56:33.934041 30582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:56:33.934671 30582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:56:33.935966 30624 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 10:56:33.941200 30582 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 10:56:33.941279 30582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260927 10:56:33.941361 30582 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 10:56:33.943013 30582 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 10:56:33.943145 30582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 10:56:33.943388 30624 tablet_bootstrap.cc:492] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec: Bootstrap starting.
I20260927 10:56:33.952642 30582 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:36069
I20260927 10:56:33.952735 30731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:36069 every 8 connection(s)
I20260927 10:56:33.953594 30582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:56:33.961217 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 30582
I20260927 10:56:33.961735 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:43669
--local_ip_for_outbound_sockets=127.14.68.2
--tserver_master_addrs=127.14.68.62:44105
--webserver_port=42189
--webserver_interface=127.14.68.2
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:41573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:56:33.963416 30624 log.cc:824] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:56:33 dist-test-slave-tz51 krb5kdc[29964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506593, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:56:33.972198 30624 tablet_bootstrap.cc:492] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:56:33.972633 30624 tablet_bootstrap.cc:492] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec: Bootstrap complete.
I20260927 10:56:33.973272 30735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:33.955747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.1:51007)
0927 10:56:33.956984 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:33.957041 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:33.958213 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:33.959416 (+ 1203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:33.959431 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:33.960009 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:56:33.960672 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:33.960698 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:33.962233 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:33.962236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:33.963158 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:33.963173 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:33.964304 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:33.964308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:33.964519 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:33.965194 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:56:33.965214 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:56:33.967073 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:56:33.969730 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:33.969735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:33.969747 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:33.970104 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:33.970504 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:33.970507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:33.970509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:33.970661 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:33.971387 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:56:33.971393 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:56:33.971674 (+ 281us) client_negotiation.cc:769] Sending connection context
0927 10:56:33.971890 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 10:56:33.972224 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"thread_start_us":131,"threads_started":1}
I20260927 10:56:33.973668 30734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:33.955674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:51007 (local address 127.14.68.62:44105)
0927 10:56:33.956115 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:33.956131 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:33.957182 (+ 1051us) server_negotiation.cc:407] Connection header received
0927 10:56:33.958664 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:33.958675 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:33.958961 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:33.959256 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:56:33.960879 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:33.962028 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:33.963329 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:33.964130 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:33.964324 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:33.967297 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:56:33.967304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:56:33.967319 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:56:33.967353 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:56:33.969590 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:33.970237 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:33.970243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:33.970248 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:33.970397 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:33.970767 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:33.970770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:33.970772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:33.971212 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:56:33.971334 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:33.972535 (+ 1201us) server_negotiation.cc:299] Negotiation successful
0927 10:56:33.972863 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":123,"threads_started":1}
I20260927 10:56:33.974011 30624 ts_tablet_manager.cc:1397] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec: Time spent bootstrapping tablet: real 0.031s user 0.025s sys 0.004s
I20260927 10:56:33.976411 30732 heartbeater.cc:347] Connected to a master server at 127.14.68.62:44105
I20260927 10:56:33.976656 30732 heartbeater.cc:464] Registering TS with master...
I20260927 10:56:33.976773 30624 raft_consensus.cc:348] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } }
I20260927 10:56:33.977203 30732 heartbeater.cc:511] Master 127.14.68.62:44105 requested a full tablet report, sending...
I20260927 10:56:33.977268 30624 raft_consensus.cc:729] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a1cea0e89b4cab91b821773778f2ec, State: Initialized, Role: FOLLOWER
I20260927 10:56:33.977798 30624 consensus_queue.cc:260] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } }
I20260927 10:56:33.978909 30624 ts_tablet_manager.cc:1428] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:56:33.980402 30544 ts_manager.cc:194] Registered new tserver with Master: 01a1cea0e89b4cab91b821773778f2ec (127.14.68.1:36069)
I20260927 10:56:33.983759 30544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:51007
I20260927 10:56:33.986541 30732 heartbeater.cc:503] Master 127.14.68.62:44105 was elected leader, sending a full tablet report...
Sep 27 10:56:33 dist-test-slave-tz51 krb5kdc[29964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506593, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for kudu/127.14.68.1@KRBTEST.COM
I20260927 10:56:33.994068 30740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:33.983166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41176 (local address 127.14.68.1:36069)
0927 10:56:33.983839 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:33.983852 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:33.983877 (+ 25us) server_negotiation.cc:407] Connection header received
0927 10:56:33.984084 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:33.984089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:33.984314 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:33.984456 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:56:33.985170 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:33.986303 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:33.987486 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:33.988053 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:33.988156 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:33.990451 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:56:33.990459 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:56:33.990471 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:56:33.990501 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:56:33.992182 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:33.992671 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:33.992684 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:33.992685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:33.992762 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:33.993047 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:33.993052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:33.993054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:33.993281 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:56:33.993418 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:33.993752 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 10:56:33.993908 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":580,"thread_start_us":119,"threads_started":1}
I20260927 10:56:33.994071 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:33.983052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:36069 (local address 127.0.0.1:41176)
0927 10:56:33.983738 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:33.983786 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:33.983920 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:33.984480 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:33.984486 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:33.984907 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:56:33.985038 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:33.985046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:33.986441 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:33.986444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:33.987351 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:33.987363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:33.988179 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:33.988183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:33.988287 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:33.988813 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:56:33.988828 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:56:33.990309 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:56:33.992297 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:33.992308 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:33.992314 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:33.992555 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:33.992856 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:33.992861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:33.992863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:33.992949 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:33.993396 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:56:33.993402 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:56:33.993528 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 10:56:33.993675 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 10:56:33.993874 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":112,"threads_started":1}
I20260927 10:56:33.995218 30649 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e79781aec5463880e711a8a0563645 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:56:13 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:41176
I20260927 10:56:33.996088 30741 tablet_replica.cc:330] stopping tablet replica
I20260927 10:56:33.996495 30741 raft_consensus.cc:2229] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:56:33.996780 30741 raft_consensus.cc:2258] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:56:33.997845 30741 ts_tablet_manager.cc:1905] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:56:34.002942 30741 ts_tablet_manager.cc:1918] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:56:34.003063 30741 log.cc:1196] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/wal/wals/37e79781aec5463880e711a8a0563645
I20260927 10:56:34.003362 30741 ts_tablet_manager.cc:1939] T 37e79781aec5463880e711a8a0563645 P 01a1cea0e89b4cab91b821773778f2ec: Deleting consensus metadata
I20260927 10:56:34.005442 30531 catalog_manager.cc:4841] TS 01a1cea0e89b4cab91b821773778f2ec (127.14.68.1:36069): tablet 37e79781aec5463880e711a8a0563645 (table test-table [id=f4ea637431e744f897d0a57bde5e779b]) successfully deleted
W20260927 10:56:34.082155 30736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:56:34.082406 30736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:56:34.082450 30736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:56:34.086519 30736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:56:34.086580 30736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:56:34.086680 30736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:56:34.091794 30736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:41573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:43669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=42189
--tserver_master_addrs=127.14.68.62:44105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:56:34.092825 30736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:56:34.093866 30736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:56:34.103502 30746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:34.103520 30747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:34.103536 30749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:56:34.103760 30736 server_base.cc:1034] running on GCE node
I20260927 10:56:34.104290 30736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:56:34.104907 30736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:56:34.106103 30736 hybrid_clock.cc:648] HybridClock initialized: now 1790506594106083 us; error 32 us; skew 500 ppm
Sep 27 10:56:34 dist-test-slave-tz51 krb5kdc[29964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506594, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:56:34.108911 30736 init.cc:377] Logged in from keytab as kudu/127.14.68.2@KRBTEST.COM (short username kudu)
I20260927 10:56:34.109835 30736 webserver.cc:460] Webserver started at http://127.14.68.2:42189/ using document root <none> and password file <none>
I20260927 10:56:34.110455 30736 fs_manager.cc:359] Metadata directory not provided
I20260927 10:56:34.110582 30736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:56:34.114581 30736 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:56:34.116603 30756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:34.117817 30736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:56:34.117945 30736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "f2712cbe30ef4aee8e52e90b65ecee5e"
format_stamp: "Formatted at 2026-09-27 10:56:13 on dist-test-slave-tz51"
server_key: "2e0f89edc428fb4575dcb5a09a905bc4"
server_key_iv: "99c327b4650d60c1ee0bcfd5f4961393"
server_key_version: "encryptionkey@0"
I20260927 10:56:34.118526 30736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:56:34.127933 30736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:56:34.130694 30736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:56:34.130887 30736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:56:34.131421 30736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:56:34.132237 30736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:56:34.132308 30736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:34.132376 30736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:56:34.132424 30736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:34.141341 30736 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:43669
I20260927 10:56:34.141378 30869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:43669 every 8 connection(s)
I20260927 10:56:34.142297 30736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:56:34.148860 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 30736
I20260927 10:56:34.149294 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:35715
--local_ip_for_outbound_sockets=127.14.68.3
--tserver_master_addrs=127.14.68.62:44105
--webserver_port=35035
--webserver_interface=127.14.68.3
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:41573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:56:34 dist-test-slave-tz51 krb5kdc[29964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506594, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:56:34.156517 30734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:34.144173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:43279 (local address 127.14.68.62:44105)
0927 10:56:34.144322 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:34.144326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:34.145335 (+ 1009us) server_negotiation.cc:407] Connection header received
0927 10:56:34.146242 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:34.146246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:34.146307 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:34.146419 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:56:34.147573 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.148187 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.149384 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.149634 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.149749 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:34.152623 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:56:34.152626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:56:34.152629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:56:34.152653 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:56:34.154243 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:34.154894 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:34.154897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:34.154898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:34.154969 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:34.155375 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:34.155377 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:34.155378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:34.155539 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:56:34.155615 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:34.156299 (+ 684us) server_negotiation.cc:299] Negotiation successful
0927 10:56:34.156402 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:56:34.157756 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:34.144530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.2:43279)
0927 10:56:34.145189 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:34.145229 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:34.146039 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:34.146595 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:34.146602 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:34.146951 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:56:34.147423 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:34.147437 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.148317 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.148321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:34.149258 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:34.149267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.149785 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.149789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:34.149966 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:34.150624 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:56:34.150640 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:56:34.152444 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:56:34.154373 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:34.154380 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:34.154397 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:34.154787 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:34.155067 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:34.155073 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:34.155076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:34.155273 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:34.155647 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:56:34.155655 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:56:34.156033 (+ 378us) client_negotiation.cc:769] Sending connection context
0927 10:56:34.156269 (+ 236us) client_negotiation.cc:240] Negotiation successful
0927 10:56:34.156628 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":97,"threads_started":1}
I20260927 10:56:34.159003 30870 heartbeater.cc:347] Connected to a master server at 127.14.68.62:44105
I20260927 10:56:34.159276 30870 heartbeater.cc:464] Registering TS with master...
I20260927 10:56:34.159909 30870 heartbeater.cc:511] Master 127.14.68.62:44105 requested a full tablet report, sending...
I20260927 10:56:34.161090 30544 ts_manager.cc:194] Registered new tserver with Master: f2712cbe30ef4aee8e52e90b65ecee5e (127.14.68.2:43669)
I20260927 10:56:34.161659 30544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:43279
W20260927 10:56:34.264429 30873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:56:34.264722 30873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:56:34.264835 30873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:56:34.268854 30873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:56:34.268939 30873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:56:34.269021 30873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:56:34.274210 30873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:41573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:35715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=35035
--tserver_master_addrs=127.14.68.62:44105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:56:34.275295 30873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:56:34.276367 30873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:56:34.286058 30879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:56:34.286229 30873 server_base.cc:1034] running on GCE node
W20260927 10:56:34.286031 30878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:34.286032 30881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:56:34.286716 30873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:56:34.287397 30873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:56:34.288570 30873 hybrid_clock.cc:648] HybridClock initialized: now 1790506594288534 us; error 50 us; skew 500 ppm
Sep 27 10:56:34 dist-test-slave-tz51 krb5kdc[29964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506594, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:56:34.291450 30873 init.cc:377] Logged in from keytab as kudu/127.14.68.3@KRBTEST.COM (short username kudu)
I20260927 10:56:34.292371 30873 webserver.cc:460] Webserver started at http://127.14.68.3:35035/ using document root <none> and password file <none>
I20260927 10:56:34.292924 30873 fs_manager.cc:359] Metadata directory not provided
I20260927 10:56:34.293028 30873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:56:34.296885 30873 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:56:34.298949 30888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:34.300351 30873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:56:34.300477 30873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "e6690d948ed24e9bb78f0ca93fca22c2"
format_stamp: "Formatted at 2026-09-27 10:56:13 on dist-test-slave-tz51"
server_key: "a25abb69c6ae4e67b62638274a11cbf1"
server_key_iv: "0cdf16de2ac65c9c72faac68d9641206"
server_key_version: "encryptionkey@0"
I20260927 10:56:34.301048 30873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:56:34.314484 30873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:56:34.317800 30873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:56:34.317994 30873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:56:34.318593 30873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:56:34.319442 30873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:56:34.319514 30873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:34.319584 30873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:56:34.319635 30873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:34.328356 30873 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:35715
I20260927 10:56:34.328382 31001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:35715 every 8 connection(s)
I20260927 10:56:34.329298 30873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:56:34.336283 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 30873
Sep 27 10:56:34 dist-test-slave-tz51 krb5kdc[29964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506594, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:56:34.343611 30734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:34.331013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:57605 (local address 127.14.68.62:44105)
0927 10:56:34.331144 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:34.331151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:34.331981 (+ 830us) server_negotiation.cc:407] Connection header received
0927 10:56:34.333257 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:34.333261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:34.333329 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:34.333425 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:56:34.334705 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.335232 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.336462 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.336721 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.336836 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:34.339738 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:56:34.339743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:56:34.339746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:56:34.339777 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:56:34.341284 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:34.341867 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:34.341870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:34.341871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:34.341941 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:34.342319 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:34.342322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:34.342324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:34.342555 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:56:34.342657 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:34.343222 (+ 565us) server_negotiation.cc:299] Negotiation successful
0927 10:56:34.343361 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:56:34.344626 31004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:34.331336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.3:57605)
0927 10:56:34.331837 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:34.331878 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:34.333026 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:34.333602 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:34.333609 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:34.334063 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:56:34.334553 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:34.334567 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.335371 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.335374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:34.336330 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:34.336340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.336913 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.336921 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:34.337133 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:34.337809 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:56:34.337824 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:56:34.339533 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:56:34.341395 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:34.341401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:34.341416 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:34.341765 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:34.342035 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:34.342038 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:34.342040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:34.342214 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:34.342673 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:56:34.342681 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:56:34.342983 (+ 302us) client_negotiation.cc:769] Sending connection context
0927 10:56:34.343178 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 10:56:34.343533 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":116,"threads_started":1}
I20260927 10:56:34.345790 31006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:34.336991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.14.68.62:44105)
0927 10:56:34.337286 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:34.337291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:34.337309 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:56:34.337452 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:34.337454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:34.337494 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:34.337571 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:56:34.338444 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.339028 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.340223 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.340626 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.340779 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:34.342339 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:56:34.342342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:56:34.342344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:56:34.342375 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:56:34.344045 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:34.344512 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:34.344515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:34.344516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:34.344603 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:34.344881 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:34.344885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:34.344886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:34.345084 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:56:34.345289 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:34.345504 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 10:56:34.345648 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":111,"threads_started":1}
I20260927 10:56:34.346027 31002 heartbeater.cc:347] Connected to a master server at 127.14.68.62:44105
I20260927 10:56:34.346338 31002 heartbeater.cc:464] Registering TS with master...
I20260927 10:56:34.346899 31002 heartbeater.cc:511] Master 127.14.68.62:44105 requested a full tablet report, sending...
I20260927 10:56:34.347882 30544 ts_manager.cc:194] Registered new tserver with Master: e6690d948ed24e9bb78f0ca93fca22c2 (127.14.68.3:35715)
I20260927 10:56:34.348415 30544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:57605
I20260927 10:56:34.359357 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:56:34.363617 31006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:34.359999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56732 (local address 127.14.68.62:44105)
0927 10:56:34.360154 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:34.360158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:34.360321 (+ 163us) server_negotiation.cc:407] Connection header received
0927 10:56:34.360503 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:34.360505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:34.360598 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:34.360703 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:56:34.360964 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.361463 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.362345 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.362579 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.362692 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:34.362990 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:56:34.363120 (+ 130us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 10:56:34.363488 (+ 368us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56732: token expired
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 10:56:34.363680 31006 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56732: token expired
W20260927 10:56:34.364006 30474 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.68.62:44105)
I20260927 10:56:34.372291 31006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:34.364359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56740 (local address 127.14.68.62:44105)
0927 10:56:34.364490 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:34.364496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:34.364512 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:56:34.364552 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:34.364557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:34.364605 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:34.364698 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:56:34.365277 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.366076 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.366982 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.367370 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.367497 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:34.368797 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:56:34.368801 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:56:34.368804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:56:34.368827 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:56:34.370728 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:34.371155 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:34.371158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:34.371159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:34.371231 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:34.371548 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:34.371550 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:34.371551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:34.371763 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:56:34.371873 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:34.372073 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 10:56:34.372167 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:56:34.374610 30544 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:56:34.378072 30544 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:56:34.385890 30649 tablet_service.cc:1467] Processing CreateTablet for tablet 13384a7a7fde4be2b8d6fae604bd24e5 (DEFAULT_TABLE table=test-table [id=608364a117814fe9a861fe2efbd80a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:56:34.386560 30649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13384a7a7fde4be2b8d6fae604bd24e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:56:34.389456 30624 tablet_bootstrap.cc:492] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec: Bootstrap starting.
I20260927 10:56:34.390206 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:34.384746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:43669 (local address 127.0.0.1:59192)
0927 10:56:34.384992 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:34.385009 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:34.385085 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:34.386263 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:34.386268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:34.386288 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:56:34.386391 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:34.386398 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.387453 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.387459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:34.388647 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:34.388657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.389779 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.389782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:34.389887 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:34.389934 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 10:56:34.389990 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 10:56:34.390078 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260927 10:56:34.390396 30624 tablet_bootstrap.cc:654] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec: Neither blocks nor log segments found. Creating new log.
I20260927 10:56:34.390793 31011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:34.384908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59192 (local address 127.14.68.2:43669)
0927 10:56:34.385729 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:34.385743 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:34.385768 (+ 25us) server_negotiation.cc:407] Connection header received
0927 10:56:34.385851 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:34.385858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:34.386105 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:34.386273 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:56:34.386515 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.387339 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.388784 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.389641 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.390400 (+ 759us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:34.390564 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:34.390608 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 10:56:34.390681 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":126,"threads_started":1}
I20260927 10:56:34.391070 31012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:34.385029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:35715 (local address 127.0.0.1:50108)
0927 10:56:34.385350 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:34.385364 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:34.385463 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:34.386519 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:34.386522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:34.386537 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:56:34.386625 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:34.386631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.387965 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.387969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:34.388979 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:34.388986 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.390614 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.390623 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:34.390752 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:34.390787 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 10:56:34.390852 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 10:56:34.390930 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":112,"threads_started":1}
I20260927 10:56:34.391888 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:34.385453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50108 (local address 127.14.68.3:35715)
0927 10:56:34.385803 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:34.385813 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:34.385830 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:56:34.385891 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:34.385895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:34.386103 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:34.386273 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:56:34.386753 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.387848 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.389103 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.390094 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.391361 (+ 1267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:34.391535 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:34.391605 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 10:56:34.391754 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":154,"threads_started":1}
I20260927 10:56:34.392022 30624 tablet_bootstrap.cc:492] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec: No bootstrap required, opened a new log
I20260927 10:56:34.392108 30624 ts_tablet_manager.cc:1397] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:56:34.392581 30624 raft_consensus.cc:348] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } }
I20260927 10:56:34.392719 30624 raft_consensus.cc:374] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:56:34.392768 30624 raft_consensus.cc:729] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a1cea0e89b4cab91b821773778f2ec, State: Initialized, Role: FOLLOWER
I20260927 10:56:34.392891 30624 consensus_queue.cc:260] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } }
I20260927 10:56:34.393179 30624 ts_tablet_manager.cc:1428] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:56:34.394014 30804 tablet_service.cc:1467] Processing CreateTablet for tablet 13384a7a7fde4be2b8d6fae604bd24e5 (DEFAULT_TABLE table=test-table [id=608364a117814fe9a861fe2efbd80a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:56:34.394658 30936 tablet_service.cc:1467] Processing CreateTablet for tablet 13384a7a7fde4be2b8d6fae604bd24e5 (DEFAULT_TABLE table=test-table [id=608364a117814fe9a861fe2efbd80a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:56:34.395170 30804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13384a7a7fde4be2b8d6fae604bd24e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:56:34.395680 30936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13384a7a7fde4be2b8d6fae604bd24e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:56:34.403252 31015 tablet_bootstrap.cc:492] T 13384a7a7fde4be2b8d6fae604bd24e5 P e6690d948ed24e9bb78f0ca93fca22c2: Bootstrap starting.
I20260927 10:56:34.403733 31016 tablet_bootstrap.cc:492] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e: Bootstrap starting.
I20260927 10:56:34.406276 31016 tablet_bootstrap.cc:654] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e: Neither blocks nor log segments found. Creating new log.
I20260927 10:56:34.406270 31015 tablet_bootstrap.cc:654] T 13384a7a7fde4be2b8d6fae604bd24e5 P e6690d948ed24e9bb78f0ca93fca22c2: Neither blocks nor log segments found. Creating new log.
I20260927 10:56:34.407182 31015 log.cc:824] T 13384a7a7fde4be2b8d6fae604bd24e5 P e6690d948ed24e9bb78f0ca93fca22c2: Log is configured to *not* fsync() on all Append() calls
I20260927 10:56:34.407182 31016 log.cc:824] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e: Log is configured to *not* fsync() on all Append() calls
I20260927 10:56:34.409266 31016 tablet_bootstrap.cc:492] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e: No bootstrap required, opened a new log
I20260927 10:56:34.409266 31015 tablet_bootstrap.cc:492] T 13384a7a7fde4be2b8d6fae604bd24e5 P e6690d948ed24e9bb78f0ca93fca22c2: No bootstrap required, opened a new log
I20260927 10:56:34.409487 31016 ts_tablet_manager.cc:1397] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:56:34.409480 31015 ts_tablet_manager.cc:1397] T 13384a7a7fde4be2b8d6fae604bd24e5 P e6690d948ed24e9bb78f0ca93fca22c2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:56:34.413517 31016 raft_consensus.cc:348] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } }
I20260927 10:56:34.413517 31015 raft_consensus.cc:348] T 13384a7a7fde4be2b8d6fae604bd24e5 P e6690d948ed24e9bb78f0ca93fca22c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } }
I20260927 10:56:34.413820 31015 raft_consensus.cc:374] T 13384a7a7fde4be2b8d6fae604bd24e5 P e6690d948ed24e9bb78f0ca93fca22c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:56:34.413820 31016 raft_consensus.cc:374] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:56:34.413883 31016 raft_consensus.cc:729] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2712cbe30ef4aee8e52e90b65ecee5e, State: Initialized, Role: FOLLOWER
I20260927 10:56:34.413883 31015 raft_consensus.cc:729] T 13384a7a7fde4be2b8d6fae604bd24e5 P e6690d948ed24e9bb78f0ca93fca22c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6690d948ed24e9bb78f0ca93fca22c2, State: Initialized, Role: FOLLOWER
I20260927 10:56:34.414361 31016 consensus_queue.cc:260] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } }
I20260927 10:56:34.414361 31015 consensus_queue.cc:260] T 13384a7a7fde4be2b8d6fae604bd24e5 P e6690d948ed24e9bb78f0ca93fca22c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } }
I20260927 10:56:34.415014 31002 heartbeater.cc:503] Master 127.14.68.62:44105 was elected leader, sending a full tablet report...
I20260927 10:56:34.415292 30870 heartbeater.cc:503] Master 127.14.68.62:44105 was elected leader, sending a full tablet report...
I20260927 10:56:34.415376 31015 ts_tablet_manager.cc:1428] T 13384a7a7fde4be2b8d6fae604bd24e5 P e6690d948ed24e9bb78f0ca93fca22c2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:56:34.415376 31016 ts_tablet_manager.cc:1428] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260927 10:56:34.454953 30733 tablet.cc:2369] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:56:34.580237 31003 tablet.cc:2369] T 13384a7a7fde4be2b8d6fae604bd24e5 P e6690d948ed24e9bb78f0ca93fca22c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:56:34.604892 31019 raft_consensus.cc:482] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:56:34.605121 31019 raft_consensus.cc:504] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } }
I20260927 10:56:34.606388 31019 leader_election.cc:288] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6690d948ed24e9bb78f0ca93fca22c2 (127.14.68.3:35715), 01a1cea0e89b4cab91b821773778f2ec (127.14.68.1:36069)
I20260927 10:56:34.611001 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:34.606671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:35715 (local address 127.14.68.2:45627)
0927 10:56:34.606788 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:34.606804 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:34.606907 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:34.607228 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:34.607231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:34.607258 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:56:34.607391 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:34.607396 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.608419 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.608421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:34.609683 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:34.609694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.610539 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.610543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:34.610666 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:34.610698 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 10:56:34.610776 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 10:56:34.610850 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"spinlock_wait_cycles":1792}
I20260927 10:56:34.611018 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:34.606777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:45627 (local address 127.14.68.3:35715)
0927 10:56:34.606973 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:34.606977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:34.606993 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:56:34.607070 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:34.607073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:34.607115 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:34.607182 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:56:34.607505 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.608314 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.609832 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.610410 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.610517 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:34.610591 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:34.610812 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 10:56:34.610894 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260927 10:56:34.611124 31022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:34.606677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:36069 (local address 127.14.68.2:44789)
0927 10:56:34.607141 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:34.607151 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:34.607263 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:34.607541 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:34.607543 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:34.607557 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:56:34.607648 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:34.607652 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.608632 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.608635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:34.609754 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:34.609764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.610705 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.610709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:34.610816 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:34.610834 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 10:56:34.610904 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 10:56:34.610959 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":116,"spinlock_wait_cycles":2304,"thread_start_us":111,"threads_started":1}
I20260927 10:56:34.611129 31021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:34.606722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:44789 (local address 127.14.68.1:36069)
0927 10:56:34.607009 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:34.607015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:34.607262 (+ 247us) server_negotiation.cc:407] Connection header received
0927 10:56:34.607348 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:34.607350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:34.607402 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:34.607526 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:56:34.607799 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.608525 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.609868 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.610585 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.610685 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:34.610842 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:34.610918 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 10:56:34.610988 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":77,"threads_started":1}
I20260927 10:56:34.611620 30686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13384a7a7fde4be2b8d6fae604bd24e5" candidate_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01a1cea0e89b4cab91b821773778f2ec" is_pre_election: true
I20260927 10:56:34.611994 30686 raft_consensus.cc:2454] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2712cbe30ef4aee8e52e90b65ecee5e in term 0.
I20260927 10:56:34.612236 30956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13384a7a7fde4be2b8d6fae604bd24e5" candidate_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" is_pre_election: true
I20260927 10:56:34.612463 30760 leader_election.cc:302] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a1cea0e89b4cab91b821773778f2ec, f2712cbe30ef4aee8e52e90b65ecee5e; no voters:
I20260927 10:56:34.612535 30956 raft_consensus.cc:2454] T 13384a7a7fde4be2b8d6fae604bd24e5 P e6690d948ed24e9bb78f0ca93fca22c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2712cbe30ef4aee8e52e90b65ecee5e in term 0.
I20260927 10:56:34.612744 31019 raft_consensus.cc:2781] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:56:34.612838 31019 raft_consensus.cc:482] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:56:34.612896 31019 raft_consensus.cc:3037] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:56:34.615356 31019 raft_consensus.cc:504] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } }
I20260927 10:56:34.615756 31019 leader_election.cc:288] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e [CANDIDATE]: Term 1 election: Requested vote from peers e6690d948ed24e9bb78f0ca93fca22c2 (127.14.68.3:35715), 01a1cea0e89b4cab91b821773778f2ec (127.14.68.1:36069)
I20260927 10:56:34.616017 30956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13384a7a7fde4be2b8d6fae604bd24e5" candidate_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6690d948ed24e9bb78f0ca93fca22c2"
I20260927 10:56:34.616117 30686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13384a7a7fde4be2b8d6fae604bd24e5" candidate_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01a1cea0e89b4cab91b821773778f2ec"
I20260927 10:56:34.616149 30956 raft_consensus.cc:3037] T 13384a7a7fde4be2b8d6fae604bd24e5 P e6690d948ed24e9bb78f0ca93fca22c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:56:34.616242 30686 raft_consensus.cc:3037] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:56:34.617415 30956 raft_consensus.cc:2454] T 13384a7a7fde4be2b8d6fae604bd24e5 P e6690d948ed24e9bb78f0ca93fca22c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2712cbe30ef4aee8e52e90b65ecee5e in term 1.
I20260927 10:56:34.617415 30686 raft_consensus.cc:2454] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2712cbe30ef4aee8e52e90b65ecee5e in term 1.
I20260927 10:56:34.617682 30760 leader_election.cc:302] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a1cea0e89b4cab91b821773778f2ec, f2712cbe30ef4aee8e52e90b65ecee5e; no voters:
I20260927 10:56:34.617861 31019 raft_consensus.cc:2781] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:56:34.618096 31019 raft_consensus.cc:686] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e [term 1 LEADER]: Becoming Leader. State: Replica: f2712cbe30ef4aee8e52e90b65ecee5e, State: Running, Role: LEADER
I20260927 10:56:34.618423 31019 consensus_queue.cc:237] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } }
I20260927 10:56:34.621526 30542 catalog_manager.cc:5495] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e reported cstate change: term changed from 0 to 1, leader changed from <none> to f2712cbe30ef4aee8e52e90b65ecee5e (127.14.68.2). New cstate: current_term: 1 leader_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } health_report { overall_health: UNKNOWN } } }
W20260927 10:56:34.643494 30871 tablet.cc:2369] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:56:34.685848 31011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:34.682288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59200 (local address 127.14.68.2:43669)
0927 10:56:34.682436 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:34.682441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:34.682457 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:56:34.682562 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:34.682565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:34.682633 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:34.682754 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:56:34.683018 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.683529 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.684410 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:34.684634 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:34.684744 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:34.685047 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:56:34.685448 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:56:34.685584 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:34.685627 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 10:56:34.685684 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:56:34.691529 30956 raft_consensus.cc:1261] T 13384a7a7fde4be2b8d6fae604bd24e5 P e6690d948ed24e9bb78f0ca93fca22c2 [term 1 FOLLOWER]: Refusing update from remote peer f2712cbe30ef4aee8e52e90b65ecee5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:56:34.691556 30686 raft_consensus.cc:1261] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [term 1 FOLLOWER]: Refusing update from remote peer f2712cbe30ef4aee8e52e90b65ecee5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:56:34.692276 31023 consensus_queue.cc:1035] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:56:34.692499 31019 consensus_queue.cc:1035] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:56:34.701929 31026 mvcc.cc:205] Tried to move back new op lower bound from 7333915011849678848 to 7333915011559714816. Current Snapshot: MvccSnapshot[applied={T|T < 7333915011849678848 or (T in {7333915011849678848})}]
I20260927 10:56:34.707329 30543 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:56:34.707496 30543 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:56:34.709771 30543 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd: Sending DeleteTablet for 3 replicas of tablet 13384a7a7fde4be2b8d6fae604bd24e5
I20260927 10:56:34.710417 30804 tablet_service.cc:1514] Processing DeleteTablet for tablet 13384a7a7fde4be2b8d6fae604bd24e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:56:34 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:59192
I20260927 10:56:34.710506 30936 tablet_service.cc:1514] Processing DeleteTablet for tablet 13384a7a7fde4be2b8d6fae604bd24e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:56:34 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:50108
I20260927 10:56:34.710871 31034 tablet_replica.cc:330] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e: stopping tablet replica
I20260927 10:56:34.711114 30649 tablet_service.cc:1514] Processing DeleteTablet for tablet 13384a7a7fde4be2b8d6fae604bd24e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:56:34 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:41176
I20260927 10:56:34.711197 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 30582
I20260927 10:56:34.711936 31034 raft_consensus.cc:2229] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:56:34.712261 31034 raft_consensus.cc:2258] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:56:34.712608 31035 tablet_replica.cc:330] T 13384a7a7fde4be2b8d6fae604bd24e5 P e6690d948ed24e9bb78f0ca93fca22c2: stopping tablet replica
I20260927 10:56:34.714205 31034 ts_tablet_manager.cc:1905] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:56:34.717005 31034 ts_tablet_manager.cc:1918] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:56:34.717108 31034 log.cc:1196] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/13384a7a7fde4be2b8d6fae604bd24e5
I20260927 10:56:34.717430 31034 ts_tablet_manager.cc:1939] T 13384a7a7fde4be2b8d6fae604bd24e5 P f2712cbe30ef4aee8e52e90b65ecee5e: Deleting consensus metadata
W20260927 10:56:34.717772 30531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.68.1:36069 (error 108)
I20260927 10:56:34.718328 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 30736
I20260927 10:56:34.718640 31035 raft_consensus.cc:2229] T 13384a7a7fde4be2b8d6fae604bd24e5 P e6690d948ed24e9bb78f0ca93fca22c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:56:34.718927 31035 raft_consensus.cc:2258] T 13384a7a7fde4be2b8d6fae604bd24e5 P e6690d948ed24e9bb78f0ca93fca22c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:56:34.720304 31035 ts_tablet_manager.cc:1905] T 13384a7a7fde4be2b8d6fae604bd24e5 P e6690d948ed24e9bb78f0ca93fca22c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:56:34.720948 31012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:34.720678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:36069 (local address 127.0.0.1:41182)
0927 10:56:34.720795 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:34.720868 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.1:36069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 10:56:34.721115 30531 catalog_manager.cc:4584] TS 01a1cea0e89b4cab91b821773778f2ec (127.14.68.1:36069): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 13384a7a7fde4be2b8d6fae604bd24e5: Network error: Client connection negotiation failed: client connection to 127.14.68.1:36069: connect: Connection refused (error 111)
I20260927 10:56:34.723160 31035 ts_tablet_manager.cc:1918] T 13384a7a7fde4be2b8d6fae604bd24e5 P e6690d948ed24e9bb78f0ca93fca22c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:56:34.723253 31035 log.cc:1196] T 13384a7a7fde4be2b8d6fae604bd24e5 P e6690d948ed24e9bb78f0ca93fca22c2: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/13384a7a7fde4be2b8d6fae604bd24e5
I20260927 10:56:34.723562 31035 ts_tablet_manager.cc:1939] T 13384a7a7fde4be2b8d6fae604bd24e5 P e6690d948ed24e9bb78f0ca93fca22c2: Deleting consensus metadata
I20260927 10:56:34.724728 30528 catalog_manager.cc:4841] TS e6690d948ed24e9bb78f0ca93fca22c2 (127.14.68.3:35715): tablet 13384a7a7fde4be2b8d6fae604bd24e5 (table test-table [id=608364a117814fe9a861fe2efbd80a70]) successfully deleted
I20260927 10:56:34.725268 31012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:34.724995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:43669 (local address 127.0.0.1:59214)
0927 10:56:34.725128 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:34.725187 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.2:43669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 10:56:34.725351 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 30873
I20260927 10:56:34.730839 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 30511
I20260927 10:56:54.737517 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:44105
--webserver_interface=127.14.68.62
--webserver_port=36065
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:41573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:44105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:56:54.849555 31037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:56:54.849831 31037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:56:54.849871 31037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:56:54.853951 31037 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 10:56:54.854000 31037 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 10:56:54.854017 31037 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 10:56:54.854030 31037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:56:54.854086 31037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:56:54.854130 31037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:56:54.854163 31037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:56:54.854198 31037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:56:54.859382 31037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:41573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:44105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:44105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=36065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:56:54.860448 31037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:56:54.861475 31037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:56:54.870232 31042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:54.870265 31043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:54.870308 31045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:56:54.870464 31037 server_base.cc:1034] running on GCE node
I20260927 10:56:54.871037 31037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:56:54.872151 31037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:56:54.873376 31037 hybrid_clock.cc:648] HybridClock initialized: now 1790506614873354 us; error 26 us; skew 500 ppm
Sep 27 10:56:54 dist-test-slave-tz51 krb5kdc[29964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506614, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:56:54.876163 31037 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:56:54.876955 31037 webserver.cc:460] Webserver started at http://127.14.68.62:36065/ using document root <none> and password file <none>
I20260927 10:56:54.877485 31037 fs_manager.cc:359] Metadata directory not provided
I20260927 10:56:54.877584 31037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:56:54.881155 31037 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 10:56:54.883116 31052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:54.884521 31037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:56:54.884667 31037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "f73ce2509eb8429eb44641ab241003fd"
format_stamp: "Formatted at 2026-09-27 10:56:12 on dist-test-slave-tz51"
server_key: "b6148b77380797fbfad61a2349d2a9e3"
server_key_iv: "9a72d82c969ccc3b1600a1cb0f2032f5"
server_key_version: "encryptionkey@0"
I20260927 10:56:54.885223 31037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:56:54.896013 31037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:56:54.898675 31037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:56:54.898882 31037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:56:54.905949 31037 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:44105
I20260927 10:56:54.906265 31104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:44105 every 8 connection(s)
I20260927 10:56:54.907063 31037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:56:54.911271 31105 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:56:54.912925 31105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd: Bootstrap starting.
I20260927 10:56:54.913856 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 31037
I20260927 10:56:54.914345 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:36069
--local_ip_for_outbound_sockets=127.14.68.1
--tserver_master_addrs=127.14.68.62:44105
--webserver_port=45737
--webserver_interface=127.14.68.1
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:41573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:56:54.931150 31105 log.cc:824] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd: Log is configured to *not* fsync() on all Append() calls
I20260927 10:56:54.942268 31105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:56:54.942670 31105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd: Bootstrap complete.
I20260927 10:56:54.947296 31105 raft_consensus.cc:348] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73ce2509eb8429eb44641ab241003fd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44105 } }
I20260927 10:56:54.947672 31105 raft_consensus.cc:729] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f73ce2509eb8429eb44641ab241003fd, State: Initialized, Role: FOLLOWER
I20260927 10:56:54.948200 31105 consensus_queue.cc:260] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73ce2509eb8429eb44641ab241003fd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44105 } }
I20260927 10:56:54.948314 31105 raft_consensus.cc:388] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:56:54.948400 31105 raft_consensus.cc:482] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:56:54.948498 31105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 2 FOLLOWER]: Advancing to term 3
I20260927 10:56:54.950295 31105 raft_consensus.cc:504] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73ce2509eb8429eb44641ab241003fd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44105 } }
I20260927 10:56:54.950619 31105 leader_election.cc:302] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f73ce2509eb8429eb44641ab241003fd; no voters:
I20260927 10:56:54.950999 31105 leader_election.cc:288] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 10:56:54.951086 31109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 3 FOLLOWER]: Leader election won for term 3
I20260927 10:56:54.951290 31109 raft_consensus.cc:686] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [term 3 LEADER]: Becoming Leader. State: Replica: f73ce2509eb8429eb44641ab241003fd, State: Running, Role: LEADER
I20260927 10:56:54.951568 31109 consensus_queue.cc:237] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73ce2509eb8429eb44641ab241003fd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44105 } }
I20260927 10:56:54.952261 31105 sys_catalog.cc:564] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:56:54.952929 31110 sys_catalog.cc:455] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f73ce2509eb8429eb44641ab241003fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73ce2509eb8429eb44641ab241003fd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44105 } } }
I20260927 10:56:54.953711 31110 sys_catalog.cc:458] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [sys.catalog]: This master's current role is: LEADER
I20260927 10:56:54.953604 31111 sys_catalog.cc:455] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f73ce2509eb8429eb44641ab241003fd. Latest consensus state: current_term: 3 leader_uuid: "f73ce2509eb8429eb44641ab241003fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73ce2509eb8429eb44641ab241003fd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44105 } } }
I20260927 10:56:54.954062 31111 sys_catalog.cc:458] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd [sys.catalog]: This master's current role is: LEADER
I20260927 10:56:54.954161 31121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:56:54.959065 31121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:56:54.959734 31121 catalog_manager.cc:1259] Loaded cluster ID: d977a7c9f9bd4a1b8140534a5f25b8d6
I20260927 10:56:54.959790 31121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:56:54.962245 31121 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:56:54.962982 31121 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd: Loaded TSK: 0
I20260927 10:56:54.963066 31121 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f73ce2509eb8429eb44641ab241003fd: Loaded TSK: 1
F20260927 10:56:54.963846 31121 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f12736009ad google::LogMessage::Fail() at ??:0
@ 0x7f127360388a google::LogMessage::SendToLog() at ??:0
@ 0x7f12736004de google::LogMessage::Flush() at ??:0
@ 0x7f1273601a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f12762c418d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f12762bcf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f127df657bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f127df3f99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f127df40178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f127df81df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f127dfa082b std::function<>::operator()() at ??:0
@ 0x7f127df3fb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f127df40f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f127df3d579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f127df81640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f12758523d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1275852c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f12758543f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1275843512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1276e876db start_thread at ??:0
@ 0x7f12720fe61f clone at ??:0
W20260927 10:56:55.061542 31107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:56:55.061869 31107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:56:55.061935 31107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:56:55.068018 31107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:56:55.068104 31107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:56:55.068187 31107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:56:55.075922 31107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:41573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:36069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=45737
--tserver_master_addrs=127.14.68.62:44105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:56:55.077138 31107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:56:55.078410 31107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:56:55.089104 31152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:55.089104 31153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:56:55.089376 31107 server_base.cc:1034] running on GCE node
W20260927 10:56:55.089409 31155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:56:55.089826 31107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:56:55.090590 31107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:56:55.091756 31107 hybrid_clock.cc:648] HybridClock initialized: now 1790506615091736 us; error 35 us; skew 500 ppm
Sep 27 10:56:55 dist-test-slave-tz51 krb5kdc[29964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506615, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:56:55.094821 31107 init.cc:377] Logged in from keytab as kudu/127.14.68.1@KRBTEST.COM (short username kudu)
I20260927 10:56:55.095649 31107 webserver.cc:460] Webserver started at http://127.14.68.1:45737/ using document root <none> and password file <none>
I20260927 10:56:55.096223 31107 fs_manager.cc:359] Metadata directory not provided
I20260927 10:56:55.096287 31107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:56:55.100176 31107 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:56:55.102449 31162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:55.103806 31107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:56:55.103929 31107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "01a1cea0e89b4cab91b821773778f2ec"
format_stamp: "Formatted at 2026-09-27 10:56:12 on dist-test-slave-tz51"
server_key: "0f10592efc4532cdcf454986e944502b"
server_key_iv: "5472ca3d2d5bf74b6d6f81878380c73c"
server_key_version: "encryptionkey@0"
I20260927 10:56:55.104468 31107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:56:55.128098 31107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:56:55.131338 31107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:56:55.131526 31107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:56:55.132162 31107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:56:55.133268 31169 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 10:56:55.138639 31107 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 10:56:55.138743 31107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 10:56:55.138835 31107 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 10:56:55.140588 31107 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 10:56:55.140658 31107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 10:56:55.140944 31169 tablet_bootstrap.cc:492] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec: Bootstrap starting.
I20260927 10:56:55.151345 31107 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:36069
I20260927 10:56:55.151619 31276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:36069 every 8 connection(s)
I20260927 10:56:55.152357 31107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:56:55.161090 31169 log.cc:824] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec: Log is configured to *not* fsync() on all Append() calls
I20260927 10:56:55.161933 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 31107
I20260927 10:56:55.162443 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:43669
--local_ip_for_outbound_sockets=127.14.68.2
--tserver_master_addrs=127.14.68.62:44105
--webserver_port=42189
--webserver_interface=127.14.68.2
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:41573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:56:55.169837 31169 tablet_bootstrap.cc:492] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:56:55.170292 31169 tablet_bootstrap.cc:492] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec: Bootstrap complete.
I20260927 10:56:55.171805 31169 ts_tablet_manager.cc:1397] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec: Time spent bootstrapping tablet: real 0.031s user 0.026s sys 0.004s
I20260927 10:56:55.175595 31169 raft_consensus.cc:348] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } }
I20260927 10:56:55.176174 31169 raft_consensus.cc:729] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a1cea0e89b4cab91b821773778f2ec, State: Initialized, Role: FOLLOWER
I20260927 10:56:55.176697 31169 consensus_queue.cc:260] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } }
I20260927 10:56:55.177717 31169 ts_tablet_manager.cc:1428] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260927 10:56:55.201241 31279 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:56:55.154270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.1:38879)
0927 10:56:55.154779 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:55.154844 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:55.155963 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:55.200494 (+ 44531us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":116,"threads_started":1}
W20260927 10:56:55.201820 31277 heartbeater.cc:644] Failed to heartbeat to 127.14.68.62:44105 (0 consecutive failures): Network error: Failed to ping master at 127.14.68.62:44105: Client connection negotiation failed: client connection to 127.14.68.62:44105: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 10:56:55.202579 31279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:55.202324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.1:37595)
0927 10:56:55.202446 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:55.202526 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 10:56:55.203294 31279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:55.203106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.1:49189)
0927 10:56:55.203205 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:55.203247 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 10:56:55.203552 31277 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:56:55.275812 31281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:56:55.276054 31281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:56:55.276096 31281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:56:55.280090 31281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:56:55.280138 31281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:56:55.280234 31281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:56:55.285245 31281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:41573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:43669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=42189
--tserver_master_addrs=127.14.68.62:44105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:56:55.286235 31281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:56:55.287215 31281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:56:55.296669 31288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:55.296744 31287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:56:55.296700 31281 server_base.cc:1034] running on GCE node
W20260927 10:56:55.296906 31290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:56:55.297317 31281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:56:55.297921 31281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:56:55.299115 31281 hybrid_clock.cc:648] HybridClock initialized: now 1790506615299104 us; error 34 us; skew 500 ppm
Sep 27 10:56:55 dist-test-slave-tz51 krb5kdc[29964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506615, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:56:55.301985 31281 init.cc:377] Logged in from keytab as kudu/127.14.68.2@KRBTEST.COM (short username kudu)
I20260927 10:56:55.302904 31281 webserver.cc:460] Webserver started at http://127.14.68.2:42189/ using document root <none> and password file <none>
I20260927 10:56:55.303478 31281 fs_manager.cc:359] Metadata directory not provided
I20260927 10:56:55.303568 31281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:56:55.307476 31281 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:56:55.309737 31297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:55.310976 31281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:56:55.311170 31281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "f2712cbe30ef4aee8e52e90b65ecee5e"
format_stamp: "Formatted at 2026-09-27 10:56:13 on dist-test-slave-tz51"
server_key: "2e0f89edc428fb4575dcb5a09a905bc4"
server_key_iv: "99c327b4650d60c1ee0bcfd5f4961393"
server_key_version: "encryptionkey@0"
I20260927 10:56:55.311780 31281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:56:55.346987 31281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:56:55.350550 31281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:56:55.350766 31281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:56:55.351402 31281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:56:55.352284 31281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:56:55.352358 31281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:55.352432 31281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:56:55.352475 31281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:55.361752 31281 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:43669
I20260927 10:56:55.361773 31410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:43669 every 8 connection(s)
I20260927 10:56:55.362771 31281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:56:55.366319 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:55.364853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.2:36165)
0927 10:56:55.365504 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:55.365620 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":114,"threads_started":1}
W20260927 10:56:55.366844 31411 heartbeater.cc:644] Failed to heartbeat to 127.14.68.62:44105 (0 consecutive failures): Network error: Failed to ping master at 127.14.68.62:44105: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
I20260927 10:56:55.367395 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:55.367192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.2:38633)
0927 10:56:55.367298 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:55.367344 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 10:56:55.368063 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:55.367848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.2:34361)
0927 10:56:55.367963 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:55.368008 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 10:56:55.368242 31411 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 10:56:55.370146 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 31281
I20260927 10:56:55.370603 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:35715
--local_ip_for_outbound_sockets=127.14.68.3
--tserver_master_addrs=127.14.68.62:44105
--webserver_port=35035
--webserver_interface=127.14.68.3
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:41573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:56:55.479948 31414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:56:55.480206 31414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:56:55.480255 31414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:56:55.484129 31414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:56:55.484176 31414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:56:55.484237 31414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:56:55.489630 31414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:41573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:35715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=35035
--tserver_master_addrs=127.14.68.62:44105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:56:55.490628 31414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:56:55.491686 31414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:56:55.501317 31420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:55.501322 31422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:55.501427 31419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:56:55.501483 31414 server_base.cc:1034] running on GCE node
I20260927 10:56:55.501981 31414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:56:55.502569 31414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:56:55.503751 31414 hybrid_clock.cc:648] HybridClock initialized: now 1790506615503719 us; error 41 us; skew 500 ppm
Sep 27 10:56:55 dist-test-slave-tz51 krb5kdc[29964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506615, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:56:55.506398 31414 init.cc:377] Logged in from keytab as kudu/127.14.68.3@KRBTEST.COM (short username kudu)
I20260927 10:56:55.507247 31414 webserver.cc:460] Webserver started at http://127.14.68.3:35035/ using document root <none> and password file <none>
I20260927 10:56:55.507880 31414 fs_manager.cc:359] Metadata directory not provided
I20260927 10:56:55.507966 31414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:56:55.511502 31414 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 10:56:55.513465 31429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:55.514690 31414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:56:55.514810 31414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "e6690d948ed24e9bb78f0ca93fca22c2"
format_stamp: "Formatted at 2026-09-27 10:56:13 on dist-test-slave-tz51"
server_key: "a25abb69c6ae4e67b62638274a11cbf1"
server_key_iv: "0cdf16de2ac65c9c72faac68d9641206"
server_key_version: "encryptionkey@0"
I20260927 10:56:55.515337 31414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:56:55.531337 31414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:56:55.534056 31414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:56:55.534258 31414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:56:55.534806 31414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:56:55.535588 31414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:56:55.535657 31414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:55.535746 31414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:56:55.535790 31414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:55.544380 31414 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:35715
I20260927 10:56:55.544405 31542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:35715 every 8 connection(s)
I20260927 10:56:55.545342 31414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:56:55.546988 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 31414
I20260927 10:56:55.549187 31546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:55.547562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.3:59525)
0927 10:56:55.548412 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:55.548512 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":184,"threads_started":1}
W20260927 10:56:55.549675 31543 heartbeater.cc:644] Failed to heartbeat to 127.14.68.62:44105 (0 consecutive failures): Network error: Failed to ping master at 127.14.68.62:44105: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
I20260927 10:56:55.550236 31546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:55.549971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.3:47349)
0927 10:56:55.550122 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:55.550175 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 10:56:55.550796 31546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:55.550619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.3:45313)
0927 10:56:55.550708 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:55.550748 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 10:56:55.550953 31543 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 10:56:56.204519 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:56.204139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.1:58505)
0927 10:56:56.204371 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:56.204423 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":70,"threads_started":1}
I20260927 10:56:56.369109 31548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:56.368745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.2:42609)
0927 10:56:56.368965 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:56.369024 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":69,"threads_started":1}
I20260927 10:56:56.551837 31549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:56.551425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.3:41773)
0927 10:56:56.551643 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:56.551741 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":67,"threads_started":1}
I20260927 10:56:56.864480 31550 raft_consensus.cc:482] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:56:56.864725 31550 raft_consensus.cc:504] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } }
I20260927 10:56:56.865831 31550 leader_election.cc:288] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2712cbe30ef4aee8e52e90b65ecee5e (127.14.68.2:43669), e6690d948ed24e9bb78f0ca93fca22c2 (127.14.68.3:35715)
Sep 27 10:56:56 dist-test-slave-tz51 krb5kdc[29964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506615, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.2@KRBTEST.COM
Sep 27 10:56:56 dist-test-slave-tz51 krb5kdc[29964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506615, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.3@KRBTEST.COM
I20260927 10:56:56.885116 31553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:56.866284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:37719 (local address 127.14.68.3:35715)
0927 10:56:56.866762 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:56.866781 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:56.866813 (+ 32us) server_negotiation.cc:407] Connection header received
0927 10:56:56.867461 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:56.867466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:56.867837 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:56.868108 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:56:56.869376 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:56.870601 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:56.871743 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:56.872209 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:56.872392 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:56.880219 (+ 7827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:56:56.880226 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:56:56.880242 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:56:56.880275 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:56:56.882327 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:56.882805 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:56.882811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:56.882816 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:56.882959 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:56.883212 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:56.883215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:56.883217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:56.883772 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:56:56.883910 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:56.884640 (+ 730us) server_negotiation.cc:299] Negotiation successful
0927 10:56:56.884920 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":59,"threads_started":1}
I20260927 10:56:56.885116 31554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:56.866344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:53365 (local address 127.14.68.2:43669)
0927 10:56:56.866726 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:56.866740 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:56.866776 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:56:56.867428 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:56.867434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:56.867837 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:56.868131 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:56:56.869391 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:56.870578 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:56.871757 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:56.872208 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:56.872391 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:56.879863 (+ 7472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:56:56.879870 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:56:56.879889 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:56:56.879927 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:56:56.882037 (+ 2110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:56.882659 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:56.882664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:56.882668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:56.882820 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:56.883213 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:56.883216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:56.883218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:56.883772 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:56:56.883910 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:56.884628 (+ 718us) server_negotiation.cc:299] Negotiation successful
0927 10:56:56.884920 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":116,"threads_started":1}
I20260927 10:56:56.885124 31551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:56.866091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:35715 (local address 127.14.68.1:37719)
0927 10:56:56.866398 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:56.866412 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:56.866571 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:56.868474 (+ 1903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:56.868476 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:56.868735 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:56:56.869209 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:56.869225 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:56.870737 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:56.870742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:56.871604 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:56.871614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:56.872334 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:56.872337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:56.872578 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:56.877972 (+ 5394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:56:56.878000 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:56:56.879979 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:56:56.882442 (+ 2463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:56.882445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:56.882447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:56.882707 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:56.883043 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:56.883046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:56.883048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:56.883121 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:56.883920 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:56:56.883927 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:56:56.884266 (+ 339us) client_negotiation.cc:769] Sending connection context
0927 10:56:56.884493 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 10:56:56.884960 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":341,"spinlock_wait_cycles":65408,"thread_start_us":78,"threads_started":1}
I20260927 10:56:56.885124 31552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:56.866079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:43669 (local address 127.14.68.1:53365)
0927 10:56:56.866373 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:56.866389 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:56.866507 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:56.868188 (+ 1681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:56.868194 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:56.868713 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:56:56.869209 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:56.869225 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:56.870735 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:56.870738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:56.871602 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:56.871612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:56.872335 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:56.872338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:56.872572 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:56.877974 (+ 5402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:56:56.877995 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:56:56.879556 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:56:56.882184 (+ 2628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:56.882189 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:56.882202 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:56.882533 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:56.882914 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:56.882916 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:56.882918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:56.883109 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:56.883920 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:56:56.883927 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:56:56.884266 (+ 339us) client_negotiation.cc:769] Sending connection context
0927 10:56:56.884493 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 10:56:56.884905 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":180,"spinlock_wait_cycles":14592,"thread_start_us":61,"threads_started":1}
W20260927 10:56:56.887897 31163 leader_election.cc:341] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6690d948ed24e9bb78f0ca93fca22c2 (127.14.68.3:35715): Not found: Tablet not found: 13384a7a7fde4be2b8d6fae604bd24e5
W20260927 10:56:56.888137 31166 leader_election.cc:341] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f2712cbe30ef4aee8e52e90b65ecee5e (127.14.68.2:43669): Not found: Tablet not found: 13384a7a7fde4be2b8d6fae604bd24e5
I20260927 10:56:56.888199 31166 leader_election.cc:302] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01a1cea0e89b4cab91b821773778f2ec; no voters: e6690d948ed24e9bb78f0ca93fca22c2, f2712cbe30ef4aee8e52e90b65ecee5e
I20260927 10:56:56.888500 31550 raft_consensus.cc:2726] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:56:57.205580 31552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:57.205283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.1:38691)
0927 10:56:57.205429 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:57.205489 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 10:56:57.370203 31555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:57.369807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.2:33535)
0927 10:56:57.370058 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:57.370108 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":71,"threads_started":1}
I20260927 10:56:57.553109 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:57.552663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.3:51709)
0927 10:56:57.552951 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:57.553012 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":78,"threads_started":1}
I20260927 10:56:58.206967 31557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:58.206505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.1:34845)
0927 10:56:58.206794 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:58.206857 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":82,"threads_started":1}
I20260927 10:56:58.371497 31558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:58.371024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.2:40147)
0927 10:56:58.371337 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:58.371404 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":98,"threads_started":1}
I20260927 10:56:58.554267 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:58.553874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.3:40165)
0927 10:56:58.554103 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:58.554171 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":67,"threads_started":1}
I20260927 10:56:59.060196 31560 raft_consensus.cc:482] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:56:59.060326 31560 raft_consensus.cc:504] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } }
I20260927 10:56:59.060823 31560 leader_election.cc:288] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2712cbe30ef4aee8e52e90b65ecee5e (127.14.68.2:43669), e6690d948ed24e9bb78f0ca93fca22c2 (127.14.68.3:35715)
W20260927 10:56:59.061574 31166 leader_election.cc:341] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f2712cbe30ef4aee8e52e90b65ecee5e (127.14.68.2:43669): Not found: Tablet not found: 13384a7a7fde4be2b8d6fae604bd24e5
W20260927 10:56:59.061654 31163 leader_election.cc:341] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6690d948ed24e9bb78f0ca93fca22c2 (127.14.68.3:35715): Not found: Tablet not found: 13384a7a7fde4be2b8d6fae604bd24e5
I20260927 10:56:59.061841 31163 leader_election.cc:302] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01a1cea0e89b4cab91b821773778f2ec; no voters: e6690d948ed24e9bb78f0ca93fca22c2, f2712cbe30ef4aee8e52e90b65ecee5e
I20260927 10:56:59.062021 31560 raft_consensus.cc:2726] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:56:59.208210 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:59.207819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.1:43195)
0927 10:56:59.208064 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:59.208113 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":93,"threads_started":1}
I20260927 10:56:59.372707 31562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:59.372336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.2:44137)
0927 10:56:59.372571 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:59.372622 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":71,"threads_started":1}
I20260927 10:56:59.555333 31563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:59.554964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.3:59919)
0927 10:56:59.555199 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:59.555248 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260927 10:57:00.209625 31564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:00.209101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.1:33421)
0927 10:57:00.209431 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:00.209484 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":107,"threads_started":1}
I20260927 10:57:00.373863 31565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:00.373509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.2:43097)
0927 10:57:00.373728 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:00.373777 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":71,"threads_started":1}
I20260927 10:57:00.556619 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:00.556201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.3:35811)
0927 10:57:00.556472 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:00.556532 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":92,"threads_started":1}
I20260927 10:57:01.210759 31567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:01.210333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.1:57549)
0927 10:57:01.210593 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:01.210659 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":84,"threads_started":1}
I20260927 10:57:01.375053 31568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:01.374629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.2:51059)
0927 10:57:01.374868 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:01.374927 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
I20260927 10:57:01.557776 31569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:01.557409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.3:36047)
0927 10:57:01.557644 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:01.557693 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
I20260927 10:57:02.177219 31570 raft_consensus.cc:482] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:57:02.177371 31570 raft_consensus.cc:504] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } }
I20260927 10:57:02.177798 31570 leader_election.cc:288] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2712cbe30ef4aee8e52e90b65ecee5e (127.14.68.2:43669), e6690d948ed24e9bb78f0ca93fca22c2 (127.14.68.3:35715)
W20260927 10:57:02.178555 31163 leader_election.cc:341] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6690d948ed24e9bb78f0ca93fca22c2 (127.14.68.3:35715): Not found: Tablet not found: 13384a7a7fde4be2b8d6fae604bd24e5
W20260927 10:57:02.178682 31166 leader_election.cc:341] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f2712cbe30ef4aee8e52e90b65ecee5e (127.14.68.2:43669): Not found: Tablet not found: 13384a7a7fde4be2b8d6fae604bd24e5
I20260927 10:57:02.178789 31166 leader_election.cc:302] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01a1cea0e89b4cab91b821773778f2ec; no voters: e6690d948ed24e9bb78f0ca93fca22c2, f2712cbe30ef4aee8e52e90b65ecee5e
I20260927 10:57:02.178969 31570 raft_consensus.cc:2726] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:57:02.212010 31571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:02.211499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.1:53857)
0927 10:57:02.211852 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:02.211917 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":106,"threads_started":1}
I20260927 10:57:02.376305 31572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:02.375862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.2:49543)
0927 10:57:02.376160 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:02.376214 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":85,"threads_started":1}
I20260927 10:57:02.558823 31573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:02.558417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.3:48641)
0927 10:57:02.558689 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:02.558738 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":71,"threads_started":1}
I20260927 10:57:03.213107 31574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:03.212688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.1:43815)
0927 10:57:03.212943 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:03.212999 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":77,"threads_started":1}
I20260927 10:57:03.377446 31575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:03.377034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.2:39601)
0927 10:57:03.377294 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:03.377347 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":78,"threads_started":1}
I20260927 10:57:03.559940 31576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:03.559503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.3:51043)
0927 10:57:03.559798 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:03.559851 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
I20260927 10:57:04.214259 31577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:04.213828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.1:57893)
0927 10:57:04.214096 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:04.214150 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":79,"threads_started":1}
I20260927 10:57:04.378626 31578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:04.378204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.2:52599)
0927 10:57:04.378471 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:04.378523 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":91,"threads_started":1}
I20260927 10:57:04.561009 31579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:04.560615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.3:56701)
0927 10:57:04.560869 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:04.560922 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":87,"threads_started":1}
I20260927 10:57:05.215510 31580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:05.215105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.1:46293)
0927 10:57:05.215343 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:05.215414 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
I20260927 10:57:05.327370 31581 raft_consensus.cc:482] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:57:05.327491 31581 raft_consensus.cc:504] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2712cbe30ef4aee8e52e90b65ecee5e" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 43669 } } peers { permanent_uuid: "e6690d948ed24e9bb78f0ca93fca22c2" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 35715 } } peers { permanent_uuid: "01a1cea0e89b4cab91b821773778f2ec" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 36069 } }
I20260927 10:57:05.327898 31581 leader_election.cc:288] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2712cbe30ef4aee8e52e90b65ecee5e (127.14.68.2:43669), e6690d948ed24e9bb78f0ca93fca22c2 (127.14.68.3:35715)
W20260927 10:57:05.328624 31163 leader_election.cc:341] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6690d948ed24e9bb78f0ca93fca22c2 (127.14.68.3:35715): Not found: Tablet not found: 13384a7a7fde4be2b8d6fae604bd24e5
W20260927 10:57:05.328773 31166 leader_election.cc:341] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f2712cbe30ef4aee8e52e90b65ecee5e (127.14.68.2:43669): Not found: Tablet not found: 13384a7a7fde4be2b8d6fae604bd24e5
I20260927 10:57:05.328845 31166 leader_election.cc:302] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01a1cea0e89b4cab91b821773778f2ec; no voters: e6690d948ed24e9bb78f0ca93fca22c2, f2712cbe30ef4aee8e52e90b65ecee5e
I20260927 10:57:05.329020 31581 raft_consensus.cc:2726] T 13384a7a7fde4be2b8d6fae604bd24e5 P 01a1cea0e89b4cab91b821773778f2ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:57:05.379662 31582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:05.379305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.2:59539)
0927 10:57:05.379529 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:05.379577 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":77,"threads_started":1}
I20260927 10:57:05.562089 31583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:05.561724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.3:39767)
0927 10:57:05.561956 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:05.562006 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":85,"threads_started":1}
I20260927 10:57:06.216630 31584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:06.216230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.1:51623)
0927 10:57:06.216482 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:06.216536 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":88,"threads_started":1}
I20260927 10:57:06.380848 31585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:06.380482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.2:43137)
0927 10:57:06.380713 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:06.380761 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":83,"threads_started":1}
I20260927 10:57:06.563194 31586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:06.562793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.3:38015)
0927 10:57:06.563034 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:06.563097 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":87,"threads_started":1}
I20260927 10:57:07.217700 31587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.217321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.1:57675)
0927 10:57:07.217544 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.217594 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":69,"threads_started":1}
I20260927 10:57:07.381960 31588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.381570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.2:56511)
0927 10:57:07.381814 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.381868 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":86,"threads_started":1}
I20260927 10:57:07.564324 31589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.563954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.3:38463)
0927 10:57:07.564188 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.564241 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":67,"threads_started":1}
I20260927 10:57:08.218828 31590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:08.218442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.1:53141)
0927 10:57:08.218683 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:08.218735 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":68,"threads_started":1}
I20260927 10:57:08.383052 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:08.382670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.2:51483)
0927 10:57:08.382919 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:08.382969 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":85,"threads_started":1}
I20260927 10:57:08.565431 31592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:08.565041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.3:36783)
0927 10:57:08.565292 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:08.565343 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":75,"threads_started":1}
I20260927 10:57:09.220103 31593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:09.219578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.1:54939)
0927 10:57:09.219941 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:09.220005 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":155,"threads_started":1}
I20260927 10:57:09.384227 31594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:09.383830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.2:51813)
0927 10:57:09.384066 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:09.384116 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":72,"threads_started":1}
I20260927 10:57:09.566469 31595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:09.566089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.3:41627)
0927 10:57:09.566330 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:09.566381 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":77,"threads_started":1}
I20260927 10:57:10.221442 31596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:10.221058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.1:58665)
0927 10:57:10.221301 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:10.221348 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
I20260927 10:57:10.385380 31597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:10.384989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44105 (local address 127.14.68.2:36129)
0927 10:57:10.385238 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:10.385288 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.62:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":85,"threads_started":1}
I20260927 10:57:10.551565 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 31107
I20260927 10:57:10.557237 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 31281
I20260927 10:57:10.562481 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 31414
2026-09-27T10:57:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59207 ms)
[----------] 30 tests from SecurityITest (241096 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:10 dist-test-slave-tz51 krb5kdc[31600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:10 dist-test-slave-tz51 krb5kdc[31600](info): set up 2 sockets
Sep 27 10:57:10 dist-test-slave-tz51 krb5kdc[31600](info): commencing operation
krb5kdc: starting...
W20260927 10:57:11.623575 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:57:11 dist-test-slave-tz51 krb5kdc[31600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:57:11.746098 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:38939
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:38939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:11.851009 31613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:11.851253 31613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:11.851292 31613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:11.855077 31613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:11.855119 31613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:11.855139 31613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:11.855177 31613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:11.855192 31613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:11.860165 31613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:38939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:38939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:11.861034 31613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:11.861927 31613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:11.869900 31619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:11.870023 31613 server_base.cc:1034] running on GCE node
W20260927 10:57:11.869906 31618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:11.870183 31621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:11.870645 31613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:11.871824 31613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:11.872979 31613 hybrid_clock.cc:648] HybridClock initialized: now 1790506631872960 us; error 31 us; skew 500 ppm
Sep 27 10:57:11 dist-test-slave-tz51 krb5kdc[31600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506631, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:11.875651 31613 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:57:11.876504 31613 webserver.cc:460] Webserver started at http://127.14.68.62:46503/ using document root <none> and password file <none>
I20260927 10:57:11.877035 31613 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:11.877122 31613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:11.877344 31613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:11.879289 31613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "08b25782fa1546ab97f699f0f756c56e"
format_stamp: "Formatted at 2026-09-27 10:57:11 on dist-test-slave-tz51"
server_key: "e66de9bb5cce99ede6155e86e3a110e2"
server_key_iv: "888c60641267ff022b3fe99f393bb3f2"
server_key_version: "encryptionkey@0"
I20260927 10:57:11.879880 31613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "08b25782fa1546ab97f699f0f756c56e"
format_stamp: "Formatted at 2026-09-27 10:57:11 on dist-test-slave-tz51"
server_key: "e66de9bb5cce99ede6155e86e3a110e2"
server_key_iv: "888c60641267ff022b3fe99f393bb3f2"
server_key_version: "encryptionkey@0"
I20260927 10:57:11.883279 31613 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:57:11.885538 31628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:11.886727 31613 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:11.886859 31613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "08b25782fa1546ab97f699f0f756c56e"
format_stamp: "Formatted at 2026-09-27 10:57:11 on dist-test-slave-tz51"
server_key: "e66de9bb5cce99ede6155e86e3a110e2"
server_key_iv: "888c60641267ff022b3fe99f393bb3f2"
server_key_version: "encryptionkey@0"
I20260927 10:57:11.886962 31613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:11.904944 31613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:11.908087 31613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:11.908290 31613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:11.914768 31613 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:38939
I20260927 10:57:11.914790 31680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:38939 every 8 connection(s)
I20260927 10:57:11.915673 31613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:11.918507 31681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:11.921860 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 31613
I20260927 10:57:11.922016 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:11.922300 14608 external_mini_cluster.cc:1428] Setting key cc47c39176e4b3c7cc3f74acc98b3ac8
I20260927 10:57:11.924304 31681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e: Bootstrap starting.
I20260927 10:57:11.926887 31681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:11.927767 31681 log.cc:824] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:11.929844 31681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e: No bootstrap required, opened a new log
Sep 27 10:57:11 dist-test-slave-tz51 krb5kdc[31600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:57:11.933764 31681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b25782fa1546ab97f699f0f756c56e" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38939 } }
I20260927 10:57:11.933983 31681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:11.934067 31681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08b25782fa1546ab97f699f0f756c56e, State: Initialized, Role: FOLLOWER
I20260927 10:57:11.934547 31681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b25782fa1546ab97f699f0f756c56e" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38939 } }
I20260927 10:57:11.934724 31681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:11.934803 31681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:11.934912 31681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:11.936041 31681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b25782fa1546ab97f699f0f756c56e" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38939 } }
I20260927 10:57:11.936432 31681 leader_election.cc:302] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08b25782fa1546ab97f699f0f756c56e; no voters:
I20260927 10:57:11.936800 31681 leader_election.cc:288] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:11.936863 31686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:11.937069 31686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e [term 1 LEADER]: Becoming Leader. State: Replica: 08b25782fa1546ab97f699f0f756c56e, State: Running, Role: LEADER
I20260927 10:57:11.937423 31686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b25782fa1546ab97f699f0f756c56e" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38939 } }
I20260927 10:57:11.937682 31684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:11.923639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45926 (local address 127.14.68.62:38939)
0927 10:57:11.924087 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:11.924105 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:11.924147 (+ 42us) server_negotiation.cc:407] Connection header received
0927 10:57:11.924943 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:11.924965 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:11.925363 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:11.925746 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:11.926543 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:11.928052 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:11.929030 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:11.929700 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:11.929859 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:11.932602 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:11.932607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:11.932620 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:11.932645 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:11.934552 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:11.935043 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:11.935048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:11.935054 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:11.935179 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:11.935467 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:11.935469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:11.935471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:11.935917 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:11.936060 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:11.936387 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 10:57:11.936699 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":107,"threads_started":1}
I20260927 10:57:11.938082 31681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:11.939088 31688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08b25782fa1546ab97f699f0f756c56e. Latest consensus state: current_term: 1 leader_uuid: "08b25782fa1546ab97f699f0f756c56e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b25782fa1546ab97f699f0f756c56e" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38939 } } }
I20260927 10:57:11.939482 31688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:11.939389 31687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08b25782fa1546ab97f699f0f756c56e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b25782fa1546ab97f699f0f756c56e" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38939 } } }
I20260927 10:57:11.940006 31687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e [sys.catalog]: This master's current role is: LEADER
W20260927 10:57:11.943008 31701 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:57:11.943068 31701 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:57:11.943143 31698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:11.943871 31698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:11.948927 31698 catalog_manager.cc:1347] Generated new cluster ID: ff126202809c4ea4b0bbc2624474cc71
I20260927 10:57:11.948984 31698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:11.962625 31698 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:11.963446 31698 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:11.969551 31698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e: Generated new TSK 0
I20260927 10:57:11.970148 31698 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
I20260927 10:57:12.003123 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:38939
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:12.114490 31707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:12.114720 31707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:12.114761 31707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:12.118747 31707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:12.118799 31707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:12.118854 31707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:57:12.123852 31707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:38939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:12.124735 31707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:12.125631 31707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:12.134876 31713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:12.134938 31707 server_base.cc:1034] running on GCE node
W20260927 10:57:12.134881 31712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:12.134892 31715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:12.135473 31707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:12.136137 31707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:12.137316 31707 hybrid_clock.cc:648] HybridClock initialized: now 1790506632137290 us; error 44 us; skew 500 ppm
Sep 27 10:57:12 dist-test-slave-tz51 krb5kdc[31600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506632, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:12.140292 31707 init.cc:377] Logged in from keytab as kudu/127.14.68.1@KRBTEST.COM (short username kudu)
I20260927 10:57:12.141103 31707 webserver.cc:460] Webserver started at http://127.14.68.1:35011/ using document root <none> and password file <none>
I20260927 10:57:12.141628 31707 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:12.141724 31707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:12.141924 31707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:12.143913 31707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "763fba500d254eee87bae1e0e2b31c12"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-tz51"
server_key: "36b2977ab36f7e2e2b0455af004a0920"
server_key_iv: "e2449e12b1ad62f117cb68e361a32326"
server_key_version: "encryptionkey@0"
I20260927 10:57:12.144467 31707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "763fba500d254eee87bae1e0e2b31c12"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-tz51"
server_key: "36b2977ab36f7e2e2b0455af004a0920"
server_key_iv: "e2449e12b1ad62f117cb68e361a32326"
server_key_version: "encryptionkey@0"
I20260927 10:57:12.147882 31707 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:57:12.150069 31722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:12.151180 31707 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:12.151304 31707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "763fba500d254eee87bae1e0e2b31c12"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-tz51"
server_key: "36b2977ab36f7e2e2b0455af004a0920"
server_key_iv: "e2449e12b1ad62f117cb68e361a32326"
server_key_version: "encryptionkey@0"
I20260927 10:57:12.151402 31707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:12.181145 31707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:12.183983 31707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:12.184182 31707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:12.184760 31707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:12.185655 31707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:12.185711 31707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:12.185760 31707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:12.185787 31707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:12.194630 31707 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:43295
I20260927 10:57:12.194653 31835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:43295 every 8 connection(s)
I20260927 10:57:12.195484 31707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:57:12.199281 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 31707
I20260927 10:57:12.199450 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:57:12.199821 14608 external_mini_cluster.cc:1428] Setting key 1c98bd5099455404012e7f852a60230a
Sep 27 10:57:12 dist-test-slave-tz51 krb5kdc[31600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506632, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.68.2@KRBTEST.COM; defaulting to no policy
I20260927 10:57:12.209923 31684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.197214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:48643 (local address 127.14.68.62:38939)
0927 10:57:12.197359 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:12.197364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:12.198639 (+ 1275us) server_negotiation.cc:407] Connection header received
0927 10:57:12.199511 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:12.199515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:12.199573 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:12.199692 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:12.200973 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.201570 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.203205 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.203582 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.203723 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.206352 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:12.206356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:12.206358 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:12.206379 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:12.207918 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:12.208403 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:12.208407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:12.208409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:12.208487 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:12.208805 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:12.208809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:12.208811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:12.209011 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:12.209111 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:12.209596 (+ 485us) server_negotiation.cc:299] Negotiation successful
0927 10:57:12.209722 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:57:12.210623 31838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.197654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:38939 (local address 127.14.68.1:48643)
0927 10:57:12.198485 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:12.198526 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:12.199265 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:12.199903 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:12.199914 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:12.200269 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:12.200830 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.200844 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.201711 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.201717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.203085 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.203095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.203790 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.203794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.203923 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.204488 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:12.204503 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:12.206189 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:12.208034 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:12.208039 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:12.208052 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:12.208303 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:12.208573 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:12.208578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:12.208580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:12.208713 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:12.209110 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:12.209116 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:12.209368 (+ 252us) client_negotiation.cc:769] Sending connection context
0927 10:57:12.209543 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 10:57:12.209845 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":520,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.2@KRBTEST.COM" created.
I20260927 10:57:12.211858 31836 heartbeater.cc:347] Connected to a master server at 127.14.68.62:38939
I20260927 10:57:12.212121 31836 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:12.212689 31836 heartbeater.cc:511] Master 127.14.68.62:38939 requested a full tablet report, sending...
I20260927 10:57:12.214226 31645 ts_manager.cc:194] Registered new tserver with Master: 763fba500d254eee87bae1e0e2b31c12 (127.14.68.1:43295)
I20260927 10:57:12.215420 31645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:48643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
I20260927 10:57:12.227530 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:38939
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:12.333743 31841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:12.333997 31841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:12.334079 31841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:12.338109 31841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:12.338192 31841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:12.338276 31841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:57:12.343580 31841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:38939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:12.344596 31841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:12.345600 31841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:12.354876 31847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:12.354940 31841 server_base.cc:1034] running on GCE node
W20260927 10:57:12.354910 31846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:12.354883 31849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:12.355564 31841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:12.356212 31841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:12.357368 31841 hybrid_clock.cc:648] HybridClock initialized: now 1790506632357355 us; error 27 us; skew 500 ppm
Sep 27 10:57:12 dist-test-slave-tz51 krb5kdc[31600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506632, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:12.360217 31841 init.cc:377] Logged in from keytab as kudu/127.14.68.2@KRBTEST.COM (short username kudu)
I20260927 10:57:12.361052 31841 webserver.cc:460] Webserver started at http://127.14.68.2:43685/ using document root <none> and password file <none>
I20260927 10:57:12.361562 31841 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:12.361617 31841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:12.361784 31841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:12.363818 31841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "6f2ffca93ae04fd9a6ab63a6908bcfac"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-tz51"
server_key: "1d69167cf297a8ec23e4d4a6b57690e8"
server_key_iv: "630c789e95af6cc8b8414602f72724ff"
server_key_version: "encryptionkey@0"
I20260927 10:57:12.364332 31841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "6f2ffca93ae04fd9a6ab63a6908bcfac"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-tz51"
server_key: "1d69167cf297a8ec23e4d4a6b57690e8"
server_key_iv: "630c789e95af6cc8b8414602f72724ff"
server_key_version: "encryptionkey@0"
I20260927 10:57:12.368045 31841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 10:57:12.370533 31856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:12.371919 31841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:57:12.372057 31841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "6f2ffca93ae04fd9a6ab63a6908bcfac"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-tz51"
server_key: "1d69167cf297a8ec23e4d4a6b57690e8"
server_key_iv: "630c789e95af6cc8b8414602f72724ff"
server_key_version: "encryptionkey@0"
I20260927 10:57:12.372174 31841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:12.390939 31841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:12.394119 31841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:12.394315 31841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:12.394881 31841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:12.395713 31841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:12.395781 31841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:12.395840 31841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:12.395885 31841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:12.404507 31841 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:34461
I20260927 10:57:12.404551 31969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:34461 every 8 connection(s)
I20260927 10:57:12.405404 31841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:57:12.413282 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 31841
I20260927 10:57:12.413445 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:57:12.413821 14608 external_mini_cluster.cc:1428] Setting key 37433c56d8bd82c609cefe8c9f5cbac2
Sep 27 10:57:12 dist-test-slave-tz51 krb5kdc[31600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506632, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:57:12.419755 31684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.407223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:35047 (local address 127.14.68.62:38939)
0927 10:57:12.407367 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:12.407371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:12.408167 (+ 796us) server_negotiation.cc:407] Connection header received
0927 10:57:12.409440 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:12.409444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:12.409504 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:12.409614 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:12.410825 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.411375 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.412654 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.412891 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.413002 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.415751 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:12.415756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:12.415759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:12.415792 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:12.417385 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:12.418010 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:12.418012 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:12.418014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:12.418088 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:12.418484 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:12.418490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:12.418495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:12.418752 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:12.418859 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:12.419346 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 10:57:12.419495 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:57:12.420418 31972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.407549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:38939 (local address 127.14.68.2:35047)
0927 10:57:12.408004 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:12.408055 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:12.409219 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:12.409785 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:12.409792 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:12.410142 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:12.410671 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.410685 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.411503 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.411507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.412522 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.412532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.413061 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.413065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.413238 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.413931 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:12.413948 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:12.415522 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:12.417507 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:12.417514 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:12.417532 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:12.417904 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:12.418185 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:12.418191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:12.418193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:12.418363 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:12.418857 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:12.418861 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:12.419086 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 10:57:12.419257 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 10:57:12.419504 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":131,"threads_started":1}
I20260927 10:57:12.421622 31970 heartbeater.cc:347] Connected to a master server at 127.14.68.62:38939
I20260927 10:57:12.421900 31970 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:12.422513 31970 heartbeater.cc:511] Master 127.14.68.62:38939 requested a full tablet report, sending...
I20260927 10:57:12.423559 31645 ts_manager.cc:194] Registered new tserver with Master: 6f2ffca93ae04fd9a6ab63a6908bcfac (127.14.68.2:34461)
WARNING: no policy specified for kudu/127.14.68.3@KRBTEST.COM; defaulting to no policy
I20260927 10:57:12.424237 31645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:35047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
I20260927 10:57:12.443354 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:38939
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:12.556545 31975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:12.556793 31975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:12.556834 31975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:12.560777 31975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:12.560830 31975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:12.560886 31975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:57:12.566021 31975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:38939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:12.566942 31975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:12.567936 31975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:12.577076 31983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:12.577243 31975 server_base.cc:1034] running on GCE node
W20260927 10:57:12.577067 31981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:12.577067 31980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:12.577834 31975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:12.578425 31975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:12.579579 31975 hybrid_clock.cc:648] HybridClock initialized: now 1790506632579566 us; error 23 us; skew 500 ppm
Sep 27 10:57:12 dist-test-slave-tz51 krb5kdc[31600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506632, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:12.582193 31975 init.cc:377] Logged in from keytab as kudu/127.14.68.3@KRBTEST.COM (short username kudu)
I20260927 10:57:12.582986 31975 webserver.cc:460] Webserver started at http://127.14.68.3:33455/ using document root <none> and password file <none>
I20260927 10:57:12.583498 31975 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:12.583554 31975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:12.583839 31975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:12.585709 31975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "944fd6c6c61f40a8b3d0fa53b07a8169"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-tz51"
server_key: "6913e6fd016b4ba34cc5b3d09042f008"
server_key_iv: "e3ebbb4b7f2101e0ae22fcdfa2ed20b2"
server_key_version: "encryptionkey@0"
I20260927 10:57:12.586279 31975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "944fd6c6c61f40a8b3d0fa53b07a8169"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-tz51"
server_key: "6913e6fd016b4ba34cc5b3d09042f008"
server_key_iv: "e3ebbb4b7f2101e0ae22fcdfa2ed20b2"
server_key_version: "encryptionkey@0"
I20260927 10:57:12.589783 31975 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:57:12.592065 31990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:12.593194 31975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:57:12.593325 31975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "944fd6c6c61f40a8b3d0fa53b07a8169"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-tz51"
server_key: "6913e6fd016b4ba34cc5b3d09042f008"
server_key_iv: "e3ebbb4b7f2101e0ae22fcdfa2ed20b2"
server_key_version: "encryptionkey@0"
I20260927 10:57:12.593431 31975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:12.622555 31975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:12.625368 31975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:12.625576 31975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:12.626178 31975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:12.626995 31975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:12.627094 31975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:12.627159 31975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:12.627187 31975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:12.635599 31975 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:33805
I20260927 10:57:12.635638 32103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:33805 every 8 connection(s)
I20260927 10:57:12.636480 31975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:57:12.642628 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 31975
I20260927 10:57:12.642778 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:57:12.643078 14608 external_mini_cluster.cc:1428] Setting key 4339ccd72b41618966ef99faba68da22
Sep 27 10:57:12 dist-test-slave-tz51 krb5kdc[31600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506632, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:57:12.650046 31684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.638201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:40637 (local address 127.14.68.62:38939)
0927 10:57:12.638373 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:12.638378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:12.639207 (+ 829us) server_negotiation.cc:407] Connection header received
0927 10:57:12.640614 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:12.640617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:12.640663 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:12.640735 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:12.641790 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.642309 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.643853 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.644151 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.644268 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.646599 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:12.646602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:12.646604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:12.646627 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:12.648176 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:12.648576 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:12.648579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:12.648580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:12.648650 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:12.648927 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:12.648929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:12.648930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:12.649087 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:12.649159 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:12.649751 (+ 592us) server_negotiation.cc:299] Negotiation successful
0927 10:57:12.649883 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:57:12.650964 32106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.638600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:38939 (local address 127.14.68.3:40637)
0927 10:57:12.639029 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:12.639081 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:12.640432 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:12.640890 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:12.640897 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:12.641231 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:12.641671 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.641684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.642426 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.642432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.643728 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.643738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.644297 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.644301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.644500 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.645033 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:12.645054 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:12.646455 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:12.648265 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:12.648269 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:12.648279 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:12.648496 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:12.648735 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:12.648737 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:12.648738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:12.648853 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:12.649169 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:12.649173 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:12.649385 (+ 212us) client_negotiation.cc:769] Sending connection context
0927 10:57:12.649672 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 10:57:12.649970 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":106,"threads_started":1}
I20260927 10:57:12.652486 32104 heartbeater.cc:347] Connected to a master server at 127.14.68.62:38939
I20260927 10:57:12.652704 32104 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:12.653218 32104 heartbeater.cc:511] Master 127.14.68.62:38939 requested a full tablet report, sending...
I20260927 10:57:12.654187 31645 ts_manager.cc:194] Registered new tserver with Master: 944fd6c6c61f40a8b3d0fa53b07a8169 (127.14.68.3:33805)
I20260927 10:57:12.654767 31645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:40637
I20260927 10:57:12.656986 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:57:12.665679 31684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.658825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33682 (local address 127.14.68.62:38939)
0927 10:57:12.658946 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:12.658949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:12.659198 (+ 249us) server_negotiation.cc:407] Connection header received
0927 10:57:12.659339 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:12.659342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:12.659380 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:12.659453 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:12.660176 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.660688 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.661494 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.661701 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.661771 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.663067 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:12.663070 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:12.663072 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:12.663089 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:12.664343 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:12.664708 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:12.664710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:12.664711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:12.664778 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:12.664994 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:12.664996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:12.664997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:12.665181 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:12.665277 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:12.665465 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 10:57:12.665549 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:57:12.668325 31645 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:57:12.671149 31645 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:57:12.685217 32118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.679205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:33805 (local address 127.0.0.1:57326)
0927 10:57:12.679906 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:12.679918 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:12.679995 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:12.680721 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:12.680728 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:12.680748 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:12.680908 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.680915 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.682433 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.682438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.683608 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.683617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.684676 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.684680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.684801 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.684839 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 10:57:12.684943 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 10:57:12.685055 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":4,"thread_start_us":101,"threads_started":1}
I20260927 10:57:12.685230 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.679817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57326 (local address 127.14.68.3:33805)
0927 10:57:12.680288 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:12.680300 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:12.680323 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:57:12.680411 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:12.680418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:12.680575 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:12.680696 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:12.681054 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.682272 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.683745 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.684543 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.684651 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.684787 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:12.684990 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 10:57:12.685087 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":155,"threads_started":1}
I20260927 10:57:12.685490 32115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.679009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:43295 (local address 127.0.0.1:48236)
0927 10:57:12.679477 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:12.679520 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:12.679640 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:12.680878 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:12.680881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:12.680904 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:12.681001 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.681006 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.682555 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.682558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.684034 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.684044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.685073 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.685077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.685162 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.685184 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:57:12.685232 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 10:57:12.685308 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":176,"thread_start_us":71,"threads_started":1}
I20260927 10:57:12.685570 32120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.679931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36188 (local address 127.14.68.2:34461)
0927 10:57:12.681005 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:12.681016 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:12.681039 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:57:12.681114 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:12.681118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:12.681261 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:12.681406 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:12.681633 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.682337 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.684107 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.684701 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.684801 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.684928 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:12.685280 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 10:57:12.685406 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":969,"thread_start_us":85,"threads_started":1}
I20260927 10:57:12.685842 32117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.679668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:34461 (local address 127.0.0.1:36188)
0927 10:57:12.680154 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:12.680166 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:12.680242 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:12.681386 (+ 1144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:12.681389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:12.681406 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:12.681505 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.681511 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.682474 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.682478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.683970 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.683982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.684985 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.684988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.685074 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.685093 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 10:57:12.685157 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 10:57:12.685210 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":166,"thread_start_us":119,"threads_started":1}
I20260927 10:57:12.686528 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.679170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48236 (local address 127.14.68.1:43295)
0927 10:57:12.680453 (+ 1283us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:12.680464 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:12.680480 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:57:12.680536 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:12.680539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:12.680713 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:12.680880 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:12.681122 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.682272 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.684165 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.684937 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.686145 (+ 1208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.686287 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:12.686326 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 10:57:12.686390 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1203,"thread_start_us":89,"threads_started":1}
I20260927 10:57:12.688369 32038 tablet_service.cc:1467] Processing CreateTablet for tablet d29031ee9a7a4c0f947bfa602d97be27 (DEFAULT_TABLE table=test-table [id=82b7ddc703474b8db8e7e92b73721497]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:12.688369 31904 tablet_service.cc:1467] Processing CreateTablet for tablet d29031ee9a7a4c0f947bfa602d97be27 (DEFAULT_TABLE table=test-table [id=82b7ddc703474b8db8e7e92b73721497]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:12.688833 31770 tablet_service.cc:1467] Processing CreateTablet for tablet d29031ee9a7a4c0f947bfa602d97be27 (DEFAULT_TABLE table=test-table [id=82b7ddc703474b8db8e7e92b73721497]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:12.689446 31904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d29031ee9a7a4c0f947bfa602d97be27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:12.689450 32038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d29031ee9a7a4c0f947bfa602d97be27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:12.689778 31770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d29031ee9a7a4c0f947bfa602d97be27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:12.695837 32121 tablet_bootstrap.cc:492] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12: Bootstrap starting.
I20260927 10:57:12.696920 32123 tablet_bootstrap.cc:492] T d29031ee9a7a4c0f947bfa602d97be27 P 944fd6c6c61f40a8b3d0fa53b07a8169: Bootstrap starting.
I20260927 10:57:12.697057 32122 tablet_bootstrap.cc:492] T d29031ee9a7a4c0f947bfa602d97be27 P 6f2ffca93ae04fd9a6ab63a6908bcfac: Bootstrap starting.
I20260927 10:57:12.698522 32121 tablet_bootstrap.cc:654] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:12.699111 32123 tablet_bootstrap.cc:654] T d29031ee9a7a4c0f947bfa602d97be27 P 944fd6c6c61f40a8b3d0fa53b07a8169: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:12.699540 32122 tablet_bootstrap.cc:654] T d29031ee9a7a4c0f947bfa602d97be27 P 6f2ffca93ae04fd9a6ab63a6908bcfac: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:12.699744 32121 log.cc:824] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:12.700369 32123 log.cc:824] T d29031ee9a7a4c0f947bfa602d97be27 P 944fd6c6c61f40a8b3d0fa53b07a8169: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:12.700369 32122 log.cc:824] T d29031ee9a7a4c0f947bfa602d97be27 P 6f2ffca93ae04fd9a6ab63a6908bcfac: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:12.701674 32121 tablet_bootstrap.cc:492] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12: No bootstrap required, opened a new log
I20260927 10:57:12.701880 32121 ts_tablet_manager.cc:1397] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:57:12.702114 32122 tablet_bootstrap.cc:492] T d29031ee9a7a4c0f947bfa602d97be27 P 6f2ffca93ae04fd9a6ab63a6908bcfac: No bootstrap required, opened a new log
I20260927 10:57:12.702306 32122 ts_tablet_manager.cc:1397] T d29031ee9a7a4c0f947bfa602d97be27 P 6f2ffca93ae04fd9a6ab63a6908bcfac: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:57:12.702569 32123 tablet_bootstrap.cc:492] T d29031ee9a7a4c0f947bfa602d97be27 P 944fd6c6c61f40a8b3d0fa53b07a8169: No bootstrap required, opened a new log
I20260927 10:57:12.702720 32123 ts_tablet_manager.cc:1397] T d29031ee9a7a4c0f947bfa602d97be27 P 944fd6c6c61f40a8b3d0fa53b07a8169: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:57:12.705031 32121 raft_consensus.cc:348] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763fba500d254eee87bae1e0e2b31c12" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 43295 } } peers { permanent_uuid: "944fd6c6c61f40a8b3d0fa53b07a8169" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 33805 } } peers { permanent_uuid: "6f2ffca93ae04fd9a6ab63a6908bcfac" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 34461 } }
I20260927 10:57:12.705281 32121 raft_consensus.cc:374] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:12.705334 32121 raft_consensus.cc:729] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 763fba500d254eee87bae1e0e2b31c12, State: Initialized, Role: FOLLOWER
I20260927 10:57:12.705763 32121 consensus_queue.cc:260] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763fba500d254eee87bae1e0e2b31c12" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 43295 } } peers { permanent_uuid: "944fd6c6c61f40a8b3d0fa53b07a8169" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 33805 } } peers { permanent_uuid: "6f2ffca93ae04fd9a6ab63a6908bcfac" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 34461 } }
I20260927 10:57:12.705973 32122 raft_consensus.cc:348] T d29031ee9a7a4c0f947bfa602d97be27 P 6f2ffca93ae04fd9a6ab63a6908bcfac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763fba500d254eee87bae1e0e2b31c12" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 43295 } } peers { permanent_uuid: "944fd6c6c61f40a8b3d0fa53b07a8169" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 33805 } } peers { permanent_uuid: "6f2ffca93ae04fd9a6ab63a6908bcfac" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 34461 } }
I20260927 10:57:12.706252 32122 raft_consensus.cc:374] T d29031ee9a7a4c0f947bfa602d97be27 P 6f2ffca93ae04fd9a6ab63a6908bcfac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:12.706265 32123 raft_consensus.cc:348] T d29031ee9a7a4c0f947bfa602d97be27 P 944fd6c6c61f40a8b3d0fa53b07a8169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763fba500d254eee87bae1e0e2b31c12" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 43295 } } peers { permanent_uuid: "944fd6c6c61f40a8b3d0fa53b07a8169" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 33805 } } peers { permanent_uuid: "6f2ffca93ae04fd9a6ab63a6908bcfac" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 34461 } }
I20260927 10:57:12.706492 32123 raft_consensus.cc:374] T d29031ee9a7a4c0f947bfa602d97be27 P 944fd6c6c61f40a8b3d0fa53b07a8169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:12.706506 31836 heartbeater.cc:503] Master 127.14.68.62:38939 was elected leader, sending a full tablet report...
I20260927 10:57:12.706557 32123 raft_consensus.cc:729] T d29031ee9a7a4c0f947bfa602d97be27 P 944fd6c6c61f40a8b3d0fa53b07a8169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 944fd6c6c61f40a8b3d0fa53b07a8169, State: Initialized, Role: FOLLOWER
I20260927 10:57:12.706790 32121 ts_tablet_manager.cc:1428] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:57:12.706324 32122 raft_consensus.cc:729] T d29031ee9a7a4c0f947bfa602d97be27 P 6f2ffca93ae04fd9a6ab63a6908bcfac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f2ffca93ae04fd9a6ab63a6908bcfac, State: Initialized, Role: FOLLOWER
I20260927 10:57:12.706955 32123 consensus_queue.cc:260] T d29031ee9a7a4c0f947bfa602d97be27 P 944fd6c6c61f40a8b3d0fa53b07a8169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763fba500d254eee87bae1e0e2b31c12" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 43295 } } peers { permanent_uuid: "944fd6c6c61f40a8b3d0fa53b07a8169" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 33805 } } peers { permanent_uuid: "6f2ffca93ae04fd9a6ab63a6908bcfac" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 34461 } }
I20260927 10:57:12.707254 32122 consensus_queue.cc:260] T d29031ee9a7a4c0f947bfa602d97be27 P 6f2ffca93ae04fd9a6ab63a6908bcfac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763fba500d254eee87bae1e0e2b31c12" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 43295 } } peers { permanent_uuid: "944fd6c6c61f40a8b3d0fa53b07a8169" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 33805 } } peers { permanent_uuid: "6f2ffca93ae04fd9a6ab63a6908bcfac" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 34461 } }
I20260927 10:57:12.707556 32104 heartbeater.cc:503] Master 127.14.68.62:38939 was elected leader, sending a full tablet report...
I20260927 10:57:12.707849 32123 ts_tablet_manager.cc:1428] T d29031ee9a7a4c0f947bfa602d97be27 P 944fd6c6c61f40a8b3d0fa53b07a8169: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:57:12.708038 32122 ts_tablet_manager.cc:1428] T d29031ee9a7a4c0f947bfa602d97be27 P 6f2ffca93ae04fd9a6ab63a6908bcfac: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:57:12.708330 31970 heartbeater.cc:503] Master 127.14.68.62:38939 was elected leader, sending a full tablet report...
I20260927 10:57:12.767266 32127 raft_consensus.cc:482] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:57:12.767447 32127 raft_consensus.cc:504] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763fba500d254eee87bae1e0e2b31c12" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 43295 } } peers { permanent_uuid: "944fd6c6c61f40a8b3d0fa53b07a8169" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 33805 } } peers { permanent_uuid: "6f2ffca93ae04fd9a6ab63a6908bcfac" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 34461 } }
I20260927 10:57:12.768581 32127 leader_election.cc:288] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 944fd6c6c61f40a8b3d0fa53b07a8169 (127.14.68.3:33805), 6f2ffca93ae04fd9a6ab63a6908bcfac (127.14.68.2:34461)
I20260927 10:57:12.772344 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.768888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:50045 (local address 127.14.68.3:33805)
0927 10:57:12.769067 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:12.769074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:12.769162 (+ 88us) server_negotiation.cc:407] Connection header received
0927 10:57:12.769345 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:12.769347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:12.769390 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:12.769471 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:12.769746 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.770246 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.771335 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.771808 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.771919 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.771979 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:12.772151 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 10:57:12.772218 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 10:57:12.772404 32120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.768996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:58309 (local address 127.14.68.2:34461)
0927 10:57:12.769121 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:12.769127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:12.769287 (+ 160us) server_negotiation.cc:407] Connection header received
0927 10:57:12.769441 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:12.769445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:12.769494 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:12.769580 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:12.769839 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.770324 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.771382 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.771848 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.771962 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.772021 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:12.772208 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 10:57:12.772274 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:57:12.772439 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.768843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:34461 (local address 127.14.68.1:58309)
0927 10:57:12.769191 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:12.769202 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:12.769323 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:12.769592 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:12.769594 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:12.769610 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:12.769729 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.769734 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.770441 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.770446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.771255 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.771264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.771967 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.771971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.772089 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.772106 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 10:57:12.772195 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 10:57:12.772273 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":27,"thread_start_us":120,"threads_started":1}
I20260927 10:57:12.772603 32130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.768764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:33805 (local address 127.14.68.1:50045)
0927 10:57:12.769065 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:12.769080 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:12.769228 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:12.769493 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:12.769496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:12.769511 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:12.769637 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.769643 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.770351 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.770358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.771212 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.771221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.771921 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.771924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.772020 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.772049 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 10:57:12.772135 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 10:57:12.772211 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260927 10:57:12.773070 31924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29031ee9a7a4c0f947bfa602d97be27" candidate_uuid: "763fba500d254eee87bae1e0e2b31c12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f2ffca93ae04fd9a6ab63a6908bcfac" is_pre_election: true
I20260927 10:57:12.773252 32058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29031ee9a7a4c0f947bfa602d97be27" candidate_uuid: "763fba500d254eee87bae1e0e2b31c12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "944fd6c6c61f40a8b3d0fa53b07a8169" is_pre_election: true
I20260927 10:57:12.773403 31924 raft_consensus.cc:2454] T d29031ee9a7a4c0f947bfa602d97be27 P 6f2ffca93ae04fd9a6ab63a6908bcfac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 763fba500d254eee87bae1e0e2b31c12 in term 0.
I20260927 10:57:12.773520 32058 raft_consensus.cc:2454] T d29031ee9a7a4c0f947bfa602d97be27 P 944fd6c6c61f40a8b3d0fa53b07a8169 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 763fba500d254eee87bae1e0e2b31c12 in term 0.
I20260927 10:57:12.773869 31726 leader_election.cc:302] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f2ffca93ae04fd9a6ab63a6908bcfac, 763fba500d254eee87bae1e0e2b31c12; no voters:
I20260927 10:57:12.774137 32127 raft_consensus.cc:2781] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:57:12.774221 32127 raft_consensus.cc:482] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:57:12.774252 32127 raft_consensus.cc:3037] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:12.775449 32127 raft_consensus.cc:504] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763fba500d254eee87bae1e0e2b31c12" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 43295 } } peers { permanent_uuid: "944fd6c6c61f40a8b3d0fa53b07a8169" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 33805 } } peers { permanent_uuid: "6f2ffca93ae04fd9a6ab63a6908bcfac" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 34461 } }
I20260927 10:57:12.775873 32127 leader_election.cc:288] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12 [CANDIDATE]: Term 1 election: Requested vote from peers 944fd6c6c61f40a8b3d0fa53b07a8169 (127.14.68.3:33805), 6f2ffca93ae04fd9a6ab63a6908bcfac (127.14.68.2:34461)
I20260927 10:57:12.776171 32058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29031ee9a7a4c0f947bfa602d97be27" candidate_uuid: "763fba500d254eee87bae1e0e2b31c12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "944fd6c6c61f40a8b3d0fa53b07a8169"
I20260927 10:57:12.776216 31924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29031ee9a7a4c0f947bfa602d97be27" candidate_uuid: "763fba500d254eee87bae1e0e2b31c12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f2ffca93ae04fd9a6ab63a6908bcfac"
I20260927 10:57:12.776312 31924 raft_consensus.cc:3037] T d29031ee9a7a4c0f947bfa602d97be27 P 6f2ffca93ae04fd9a6ab63a6908bcfac [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:12.776314 32058 raft_consensus.cc:3037] T d29031ee9a7a4c0f947bfa602d97be27 P 944fd6c6c61f40a8b3d0fa53b07a8169 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:12.777637 31924 raft_consensus.cc:2454] T d29031ee9a7a4c0f947bfa602d97be27 P 6f2ffca93ae04fd9a6ab63a6908bcfac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 763fba500d254eee87bae1e0e2b31c12 in term 1.
I20260927 10:57:12.777633 32058 raft_consensus.cc:2454] T d29031ee9a7a4c0f947bfa602d97be27 P 944fd6c6c61f40a8b3d0fa53b07a8169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 763fba500d254eee87bae1e0e2b31c12 in term 1.
I20260927 10:57:12.777952 31725 leader_election.cc:302] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 763fba500d254eee87bae1e0e2b31c12, 944fd6c6c61f40a8b3d0fa53b07a8169; no voters:
I20260927 10:57:12.778153 32127 raft_consensus.cc:2781] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:12.778357 32127 raft_consensus.cc:686] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12 [term 1 LEADER]: Becoming Leader. State: Replica: 763fba500d254eee87bae1e0e2b31c12, State: Running, Role: LEADER
I20260927 10:57:12.778635 32127 consensus_queue.cc:237] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763fba500d254eee87bae1e0e2b31c12" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 43295 } } peers { permanent_uuid: "944fd6c6c61f40a8b3d0fa53b07a8169" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 33805 } } peers { permanent_uuid: "6f2ffca93ae04fd9a6ab63a6908bcfac" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 34461 } }
I20260927 10:57:12.781778 31644 catalog_manager.cc:5495] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 763fba500d254eee87bae1e0e2b31c12 (127.14.68.1). New cstate: current_term: 1 leader_uuid: "763fba500d254eee87bae1e0e2b31c12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763fba500d254eee87bae1e0e2b31c12" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 43295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "944fd6c6c61f40a8b3d0fa53b07a8169" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f2ffca93ae04fd9a6ab63a6908bcfac" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 34461 } health_report { overall_health: UNKNOWN } } }
I20260927 10:57:12.807199 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.804065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48238 (local address 127.14.68.1:43295)
0927 10:57:12.804186 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:12.804189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:12.804216 (+ 27us) server_negotiation.cc:407] Connection header received
0927 10:57:12.804368 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:12.804370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:12.804443 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:12.804526 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:57:12.804725 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.805196 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.805996 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.806207 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.806279 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.806524 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:57:12.806908 (+ 384us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:57:12.806993 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:12.807052 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 10:57:12.807097 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:57:12.813341 32058 raft_consensus.cc:1261] T d29031ee9a7a4c0f947bfa602d97be27 P 944fd6c6c61f40a8b3d0fa53b07a8169 [term 1 FOLLOWER]: Refusing update from remote peer 763fba500d254eee87bae1e0e2b31c12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:12.813341 31924 raft_consensus.cc:1261] T d29031ee9a7a4c0f947bfa602d97be27 P 6f2ffca93ae04fd9a6ab63a6908bcfac [term 1 FOLLOWER]: Refusing update from remote peer 763fba500d254eee87bae1e0e2b31c12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:12.814062 32127 consensus_queue.cc:1035] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "944fd6c6c61f40a8b3d0fa53b07a8169" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:12.814275 32132 consensus_queue.cc:1035] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f2ffca93ae04fd9a6ab63a6908bcfac" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 34461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:12.823482 32135 mvcc.cc:205] Tried to move back new op lower bound from 7333915167995260928 to 7333915167863717888. Current Snapshot: MvccSnapshot[applied={T|T < 7333915167995260928 or (T in {7333915167995260928})}]
I20260927 10:57:12.823698 32136 mvcc.cc:205] Tried to move back new op lower bound from 7333915167995260928 to 7333915167863717888. Current Snapshot: MvccSnapshot[applied={T|T < 7333915167995260928}]
I20260927 10:57:12.830667 31644 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:57:12.830847 31644 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:57:12.833079 31644 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08b25782fa1546ab97f699f0f756c56e: Sending DeleteTablet for 3 replicas of tablet d29031ee9a7a4c0f947bfa602d97be27
I20260927 10:57:12.833850 32038 tablet_service.cc:1514] Processing DeleteTablet for tablet d29031ee9a7a4c0f947bfa602d97be27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:12 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:57326
I20260927 10:57:12.833894 31770 tablet_service.cc:1514] Processing DeleteTablet for tablet d29031ee9a7a4c0f947bfa602d97be27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:12 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:48236
I20260927 10:57:12.834352 32144 tablet_replica.cc:330] T d29031ee9a7a4c0f947bfa602d97be27 P 944fd6c6c61f40a8b3d0fa53b07a8169: stopping tablet replica
I20260927 10:57:12.834335 31904 tablet_service.cc:1514] Processing DeleteTablet for tablet d29031ee9a7a4c0f947bfa602d97be27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:12 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:36188
I20260927 10:57:12.834666 32144 raft_consensus.cc:2229] T d29031ee9a7a4c0f947bfa602d97be27 P 944fd6c6c61f40a8b3d0fa53b07a8169 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:57:12.834824 32146 tablet_replica.cc:330] T d29031ee9a7a4c0f947bfa602d97be27 P 6f2ffca93ae04fd9a6ab63a6908bcfac: stopping tablet replica
I20260927 10:57:12.834983 32144 raft_consensus.cc:2258] T d29031ee9a7a4c0f947bfa602d97be27 P 944fd6c6c61f40a8b3d0fa53b07a8169 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:12.835207 32146 raft_consensus.cc:2229] T d29031ee9a7a4c0f947bfa602d97be27 P 6f2ffca93ae04fd9a6ab63a6908bcfac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:57:12.835377 32145 tablet_replica.cc:330] T d29031ee9a7a4c0f947bfa602d97be27 P 763fba500d254eee87bae1e0e2b31c12: stopping tablet replica
I20260927 10:57:12.835505 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 31707
I20260927 10:57:12.837708 32144 ts_tablet_manager.cc:1905] T d29031ee9a7a4c0f947bfa602d97be27 P 944fd6c6c61f40a8b3d0fa53b07a8169: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:57:12.840265 32144 ts_tablet_manager.cc:1918] T d29031ee9a7a4c0f947bfa602d97be27 P 944fd6c6c61f40a8b3d0fa53b07a8169: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:57:12.840341 32144 log.cc:1196] T d29031ee9a7a4c0f947bfa602d97be27 P 944fd6c6c61f40a8b3d0fa53b07a8169: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/d29031ee9a7a4c0f947bfa602d97be27
I20260927 10:57:12.840636 32144 ts_tablet_manager.cc:1939] T d29031ee9a7a4c0f947bfa602d97be27 P 944fd6c6c61f40a8b3d0fa53b07a8169: Deleting consensus metadata
I20260927 10:57:12.841751 32146 raft_consensus.cc:2258] T d29031ee9a7a4c0f947bfa602d97be27 P 6f2ffca93ae04fd9a6ab63a6908bcfac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:12.841720 31631 catalog_manager.cc:4841] TS 944fd6c6c61f40a8b3d0fa53b07a8169 (127.14.68.3:33805): tablet d29031ee9a7a4c0f947bfa602d97be27 (table test-table [id=82b7ddc703474b8db8e7e92b73721497]) successfully deleted
W20260927 10:57:12.842288 31632 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.68.1:43295 (error 108)
I20260927 10:57:12.842554 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 31841
I20260927 10:57:12.843729 32117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.843385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:43295 (local address 127.0.0.1:48244)
0927 10:57:12.843531 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:12.843606 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.1:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 10:57:12.843953 31632 catalog_manager.cc:4584] TS 763fba500d254eee87bae1e0e2b31c12 (127.14.68.1:43295): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d29031ee9a7a4c0f947bfa602d97be27: Network error: Client connection negotiation failed: client connection to 127.14.68.1:43295: connect: Connection refused (error 111)
I20260927 10:57:12.847595 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 31975
I20260927 10:57:12.848456 32117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.847666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:34461 (local address 127.0.0.1:36200)
0927 10:57:12.848344 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:12.848389 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.2:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592}
I20260927 10:57:12.852945 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 31613
2026-09-27T10:57:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2290 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T10:57:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:12Z Disabled control of system clock
I20260927 10:57:12.882542 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:41241
--webserver_interface=127.14.68.62
--webserver_port=0
--builtin_ntp_servers=127.14.68.20:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:41241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:57:12.989454 32151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:12.989710 32151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:12.989745 32151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:12.993825 32151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:12.993877 32151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:12.993892 32151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:12.993909 32151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:12.993969 32151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:12.998711 32151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:41241
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:41241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:12.999815 32151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:13.001214 32151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:13.009881 32159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:13.009918 32157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:13.009892 32156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:13.010182 32151 server_base.cc:1034] running on GCE node
I20260927 10:57:13.010775 32151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:13.011828 32151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:13.012988 32151 hybrid_clock.cc:648] HybridClock initialized: now 1790506633012976 us; error 26 us; skew 500 ppm
I20260927 10:57:13.014907 32151 webserver.cc:460] Webserver started at http://127.14.68.62:37855/ using document root <none> and password file <none>
I20260927 10:57:13.015481 32151 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:13.015543 32151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:13.015849 32151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:13.017961 32151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "15fc7b6ab71e468faa8227fb15e6e228"
format_stamp: "Formatted at 2026-09-27 10:57:13 on dist-test-slave-tz51"
server_key: "30b2955de862cc2fdce8f7953571e493"
server_key_iv: "a1b158ec4fb33388e82b37e95fafe657"
server_key_version: "encryptionkey@0"
I20260927 10:57:13.018524 32151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "15fc7b6ab71e468faa8227fb15e6e228"
format_stamp: "Formatted at 2026-09-27 10:57:13 on dist-test-slave-tz51"
server_key: "30b2955de862cc2fdce8f7953571e493"
server_key_iv: "a1b158ec4fb33388e82b37e95fafe657"
server_key_version: "encryptionkey@0"
I20260927 10:57:13.022332 32151 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260927 10:57:13.024984 32165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:13.026228 32151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:57:13.026379 32151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "15fc7b6ab71e468faa8227fb15e6e228"
format_stamp: "Formatted at 2026-09-27 10:57:13 on dist-test-slave-tz51"
server_key: "30b2955de862cc2fdce8f7953571e493"
server_key_iv: "a1b158ec4fb33388e82b37e95fafe657"
server_key_version: "encryptionkey@0"
I20260927 10:57:13.026494 32151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:13.038801 32151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:13.039438 32151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:13.039636 32151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:13.046203 32151 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:41241
I20260927 10:57:13.046214 32217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:41241 every 8 connection(s)
I20260927 10:57:13.047083 32151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:13.047941 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 32151
I20260927 10:57:13.048096 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:13.048389 14608 external_mini_cluster.cc:1428] Setting key 1a98bf77c248e605f6c2ddbf1f5bceb9
I20260927 10:57:13.050457 32218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:13.055573 32218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228: Bootstrap starting.
I20260927 10:57:13.057175 32221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.049709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35992 (local address 127.14.68.62:41241)
0927 10:57:13.050328 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:13.050343 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:13.050375 (+ 32us) server_negotiation.cc:407] Connection header received
0927 10:57:13.051056 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:13.051072 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:13.051431 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:13.051801 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:13.052204 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.053129 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.054017 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.054481 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.054654 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.055507 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:13.055511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:13.055528 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:13.055978 (+ 450us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:13.056035 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:13.056057 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:13.056165 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:13.056278 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 10:57:13.056530 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":130,"threads_started":1}
I20260927 10:57:13.057792 32218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:13.058640 32218 log.cc:824] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:13.060588 32218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228: No bootstrap required, opened a new log
I20260927 10:57:13.064110 32218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15fc7b6ab71e468faa8227fb15e6e228" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41241 } }
I20260927 10:57:13.064327 32218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:13.064363 32218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15fc7b6ab71e468faa8227fb15e6e228, State: Initialized, Role: FOLLOWER
I20260927 10:57:13.064783 32218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15fc7b6ab71e468faa8227fb15e6e228" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41241 } }
I20260927 10:57:13.064951 32218 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:13.065028 32218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:13.065136 32218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:13.065972 32218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15fc7b6ab71e468faa8227fb15e6e228" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41241 } }
I20260927 10:57:13.066326 32218 leader_election.cc:302] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15fc7b6ab71e468faa8227fb15e6e228; no voters:
I20260927 10:57:13.066624 32218 leader_election.cc:288] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:13.066748 32223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:13.066977 32223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228 [term 1 LEADER]: Becoming Leader. State: Replica: 15fc7b6ab71e468faa8227fb15e6e228, State: Running, Role: LEADER
I20260927 10:57:13.067283 32223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15fc7b6ab71e468faa8227fb15e6e228" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41241 } }
I20260927 10:57:13.067780 32218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:13.068862 32225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15fc7b6ab71e468faa8227fb15e6e228. Latest consensus state: current_term: 1 leader_uuid: "15fc7b6ab71e468faa8227fb15e6e228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15fc7b6ab71e468faa8227fb15e6e228" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41241 } } }
I20260927 10:57:13.068990 32225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:13.069198 32224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15fc7b6ab71e468faa8227fb15e6e228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15fc7b6ab71e468faa8227fb15e6e228" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41241 } } }
I20260927 10:57:13.069342 32224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:13.069356 32232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:13.072058 32232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:13.077028 32232 catalog_manager.cc:1347] Generated new cluster ID: a558c9c549b64a50a20bd5694903ba60
I20260927 10:57:13.077085 32232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:13.090725 32232 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:13.091948 32232 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:13.099385 32232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228: Generated new TSK 0
I20260927 10:57:13.100152 32232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:57:13.111008 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:41241
--builtin_ntp_servers=127.14.68.20:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:57:13.217214 32242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:13.217478 32242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:13.217542 32242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:13.221480 32242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:13.221546 32242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:13.221625 32242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:57:13.226287 32242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:41241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:13.227262 32242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:13.228655 32242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:13.238178 32248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:13.238209 32250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:13.238251 32242 server_base.cc:1034] running on GCE node
W20260927 10:57:13.238201 32247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:13.238821 32242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:13.239426 32242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:13.240607 32242 hybrid_clock.cc:648] HybridClock initialized: now 1790506633240574 us; error 45 us; skew 500 ppm
I20260927 10:57:13.242364 32242 webserver.cc:460] Webserver started at http://127.14.68.1:37695/ using document root <none> and password file <none>
I20260927 10:57:13.242923 32242 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:13.243010 32242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:13.243232 32242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:13.245270 32242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "a6a2101b239246bcb948bb20853b507a"
format_stamp: "Formatted at 2026-09-27 10:57:13 on dist-test-slave-tz51"
server_key: "82c05a328c329ba1cc883ec2d9bcc823"
server_key_iv: "990e3c10bd83277e5a80d4293763d047"
server_key_version: "encryptionkey@0"
I20260927 10:57:13.245836 32242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "a6a2101b239246bcb948bb20853b507a"
format_stamp: "Formatted at 2026-09-27 10:57:13 on dist-test-slave-tz51"
server_key: "82c05a328c329ba1cc883ec2d9bcc823"
server_key_iv: "990e3c10bd83277e5a80d4293763d047"
server_key_version: "encryptionkey@0"
I20260927 10:57:13.249357 32242 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:57:13.251538 32256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:13.252703 32242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:13.252831 32242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "a6a2101b239246bcb948bb20853b507a"
format_stamp: "Formatted at 2026-09-27 10:57:13 on dist-test-slave-tz51"
server_key: "82c05a328c329ba1cc883ec2d9bcc823"
server_key_iv: "990e3c10bd83277e5a80d4293763d047"
server_key_version: "encryptionkey@0"
I20260927 10:57:13.252935 32242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:13.274525 32242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:13.275154 32242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:13.275339 32242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:13.275955 32242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:13.276836 32242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:13.276914 32242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:13.276983 32242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:13.277019 32242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:13.285141 32242 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:33223
I20260927 10:57:13.285203 32369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:33223 every 8 connection(s)
I20260927 10:57:13.286020 32242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:57:13.286984 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 32242
I20260927 10:57:13.287124 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:57:13.287446 14608 external_mini_cluster.cc:1428] Setting key a8ea7018a618b18be6a214e8f396e209
I20260927 10:57:13.289505 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:41241
--builtin_ntp_servers=127.14.68.20:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:57:13.295310 32221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.287928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:44847 (local address 127.14.68.62:41241)
0927 10:57:13.288077 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:13.288081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:13.289246 (+ 1165us) server_negotiation.cc:407] Connection header received
0927 10:57:13.290673 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:13.290676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:13.290733 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:13.290814 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:13.291654 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.292229 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.293409 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.293642 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.293754 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.294799 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:13.294803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:13.294806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:13.294817 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:13.294838 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:13.294849 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:13.294928 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:13.295084 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 10:57:13.295169 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:57:13.296132 32372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.288382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:41241 (local address 127.14.68.1:44847)
0927 10:57:13.289108 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:13.289149 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:13.290471 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:13.290974 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:13.290980 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:13.291061 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:13.291512 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.291525 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.292356 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.292360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.293294 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.293304 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.293787 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.293791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.293946 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.294550 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:57:13.294551 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:13.294588 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:57:13.294652 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:13.294941 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:13.294954 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:13.294964 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 10:57:13.295083 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 10:57:13.295337 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":106,"threads_started":1}
I20260927 10:57:13.297302 32370 heartbeater.cc:347] Connected to a master server at 127.14.68.62:41241
I20260927 10:57:13.297537 32370 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:13.298009 32370 heartbeater.cc:511] Master 127.14.68.62:41241 requested a full tablet report, sending...
I20260927 10:57:13.299604 32182 ts_manager.cc:194] Registered new tserver with Master: a6a2101b239246bcb948bb20853b507a (127.14.68.1:33223)
I20260927 10:57:13.301116 32182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.68.1:44847
W20260927 10:57:13.394568 32373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:13.394779 32373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:13.394814 32373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:13.398592 32373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:13.398630 32373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:13.398679 32373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:57:13.403081 32373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:41241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:13.403966 32373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:13.405151 32373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:13.414167 32378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:13.414186 32379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:13.414199 32381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:13.414364 32373 server_base.cc:1034] running on GCE node
I20260927 10:57:13.414772 32373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:13.415397 32373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:13.416569 32373 hybrid_clock.cc:648] HybridClock initialized: now 1790506633416536 us; error 45 us; skew 500 ppm
I20260927 10:57:13.418241 32373 webserver.cc:460] Webserver started at http://127.14.68.2:43615/ using document root <none> and password file <none>
I20260927 10:57:13.418741 32373 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:13.418798 32373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:13.418951 32373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:13.420909 32373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "ec2622a251784b229c8d177b5a756f47"
format_stamp: "Formatted at 2026-09-27 10:57:13 on dist-test-slave-tz51"
server_key: "573911b0de290e18802d4ed5f1c70eec"
server_key_iv: "23aadebdcad8bc66b9ac03655e732c89"
server_key_version: "encryptionkey@0"
I20260927 10:57:13.421399 32373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "ec2622a251784b229c8d177b5a756f47"
format_stamp: "Formatted at 2026-09-27 10:57:13 on dist-test-slave-tz51"
server_key: "573911b0de290e18802d4ed5f1c70eec"
server_key_iv: "23aadebdcad8bc66b9ac03655e732c89"
server_key_version: "encryptionkey@0"
I20260927 10:57:13.424752 32373 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 10:57:13.426942 32387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:13.428130 32373 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:57:13.428260 32373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "ec2622a251784b229c8d177b5a756f47"
format_stamp: "Formatted at 2026-09-27 10:57:13 on dist-test-slave-tz51"
server_key: "573911b0de290e18802d4ed5f1c70eec"
server_key_iv: "23aadebdcad8bc66b9ac03655e732c89"
server_key_version: "encryptionkey@0"
I20260927 10:57:13.428363 32373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:13.457719 32373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:13.458436 32373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:13.458632 32373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:13.459227 32373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:13.460209 32373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:13.460279 32373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:13.460352 32373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:13.460395 32373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:13.468995 32373 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:44597
I20260927 10:57:13.469049 32500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:44597 every 8 connection(s)
I20260927 10:57:13.469897 32373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:57:13.475879 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 32373
I20260927 10:57:13.476007 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:57:13.476303 14608 external_mini_cluster.cc:1428] Setting key 7d133b9af4032432aa0764ffdbed24c6
I20260927 10:57:13.478358 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:41241
--builtin_ntp_servers=127.14.68.20:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:57:13.479067 32221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.471662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:42069 (local address 127.14.68.62:41241)
0927 10:57:13.471836 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:13.471839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:13.472810 (+ 971us) server_negotiation.cc:407] Connection header received
0927 10:57:13.474114 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:13.474117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:13.474162 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:13.474234 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:13.475071 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.475570 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.477036 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.477279 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.477403 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.478433 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:13.478437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:13.478440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:13.478472 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:13.478494 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:13.478504 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:13.478572 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:13.478778 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 10:57:13.478855 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 10:57:13.479889 32503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.472089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:41241 (local address 127.14.68.2:42069)
0927 10:57:13.472687 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:13.472725 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:13.473931 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:13.474404 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:13.474411 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:13.474492 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:13.474954 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.474965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.475778 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.475781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.476913 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.476922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.477424 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.477428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.477596 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.478191 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:57:13.478193 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:13.478231 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:57:13.478289 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:13.478610 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:13.478619 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:13.478624 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 10:57:13.478736 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 10:57:13.478945 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":103,"threads_started":1}
I20260927 10:57:13.481029 32501 heartbeater.cc:347] Connected to a master server at 127.14.68.62:41241
I20260927 10:57:13.481252 32501 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:13.481722 32501 heartbeater.cc:511] Master 127.14.68.62:41241 requested a full tablet report, sending...
I20260927 10:57:13.482772 32182 ts_manager.cc:194] Registered new tserver with Master: ec2622a251784b229c8d177b5a756f47 (127.14.68.2:44597)
I20260927 10:57:13.483320 32182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.68.2:42069
W20260927 10:57:13.586135 32504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:13.586393 32504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:13.586467 32504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:13.590420 32504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:13.590493 32504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:13.590579 32504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:57:13.595216 32504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:41241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:13.596271 32504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:13.597570 32504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:13.606633 32512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:13.606670 32509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:13.606803 32504 server_base.cc:1034] running on GCE node
W20260927 10:57:13.606647 32510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:13.607246 32504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:13.607872 32504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:13.609047 32504 hybrid_clock.cc:648] HybridClock initialized: now 1790506633609028 us; error 32 us; skew 500 ppm
I20260927 10:57:13.610766 32504 webserver.cc:460] Webserver started at http://127.14.68.3:46241/ using document root <none> and password file <none>
I20260927 10:57:13.611302 32504 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:13.611393 32504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:13.611604 32504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:13.613528 32504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "af32407b8e90478f9c1203322448e74a"
format_stamp: "Formatted at 2026-09-27 10:57:13 on dist-test-slave-tz51"
server_key: "b92339d9780404b8712ccbb6d6459a9f"
server_key_iv: "1bbad6e4296e83c18c965a0146457785"
server_key_version: "encryptionkey@0"
I20260927 10:57:13.614102 32504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "af32407b8e90478f9c1203322448e74a"
format_stamp: "Formatted at 2026-09-27 10:57:13 on dist-test-slave-tz51"
server_key: "b92339d9780404b8712ccbb6d6459a9f"
server_key_iv: "1bbad6e4296e83c18c965a0146457785"
server_key_version: "encryptionkey@0"
I20260927 10:57:13.617506 32504 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:57:13.619881 32518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:13.621117 32504 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:57:13.621238 32504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "af32407b8e90478f9c1203322448e74a"
format_stamp: "Formatted at 2026-09-27 10:57:13 on dist-test-slave-tz51"
server_key: "b92339d9780404b8712ccbb6d6459a9f"
server_key_iv: "1bbad6e4296e83c18c965a0146457785"
server_key_version: "encryptionkey@0"
I20260927 10:57:13.621347 32504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:13.633900 32504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:13.634509 32504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:13.634706 32504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:13.635245 32504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:13.636138 32504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:13.636184 32504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:13.636220 32504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:13.636240 32504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:13.644330 32504 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:45123
I20260927 10:57:13.644361 32631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:45123 every 8 connection(s)
I20260927 10:57:13.645133 32504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:57:13.653250 32221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.646701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:33339 (local address 127.14.68.62:41241)
0927 10:57:13.646858 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:13.646863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:13.647858 (+ 995us) server_negotiation.cc:407] Connection header received
0927 10:57:13.649123 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:13.649125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:13.649175 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:13.649247 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:13.650025 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.650530 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.651552 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.651824 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.651900 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.652784 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:13.652788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:13.652790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:13.652800 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:13.652816 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:13.652824 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:13.652878 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:13.653038 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 10:57:13.653095 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:57:13.654232 32634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.647088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:41241 (local address 127.14.68.3:33339)
0927 10:57:13.647677 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:13.647766 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:13.648952 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:13.649395 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:13.649401 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:13.649480 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:13.649915 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.649925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.650627 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.650629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.651457 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.651465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.651939 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.651942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.652066 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.652562 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:57:13.652563 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:13.652598 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:57:13.652653 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:13.652892 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:13.652898 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:13.652901 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 10:57:13.653003 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 10:57:13.653287 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":105,"threads_started":1}
I20260927 10:57:13.654788 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 32504
I20260927 10:57:13.654906 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:57:13.655198 14608 external_mini_cluster.cc:1428] Setting key 930913f3522e2e925b06e19cfc6fb0b5
I20260927 10:57:13.655544 32632 heartbeater.cc:347] Connected to a master server at 127.14.68.62:41241
I20260927 10:57:13.655791 32632 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:13.656347 32632 heartbeater.cc:511] Master 127.14.68.62:41241 requested a full tablet report, sending...
I20260927 10:57:13.657449 32182 ts_manager.cc:194] Registered new tserver with Master: af32407b8e90478f9c1203322448e74a (127.14.68.3:45123)
I20260927 10:57:13.657966 32182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.68.3:33339
I20260927 10:57:13.668987 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:57:13.674122 32221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.670877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36000 (local address 127.14.68.62:41241)
0927 10:57:13.670995 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:13.670998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:13.671129 (+ 131us) server_negotiation.cc:407] Connection header received
0927 10:57:13.671233 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:13.671235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:13.671269 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:13.671331 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:13.671537 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.672077 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.672837 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.673032 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.673103 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.673808 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:13.673811 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:13.673813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:13.673825 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:13.673842 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:13.673849 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:13.673905 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:13.673982 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 10:57:13.674024 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:57:13.676695 32182 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:57:13.679545 32182 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:57:13.693087 32644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.687396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:45123 (local address 127.0.0.1:53270)
0927 10:57:13.688218 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:13.688232 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:13.688309 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:13.689028 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:13.689034 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:13.689054 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:13.689218 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.689227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.690305 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.690309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.691465 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.691474 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.692552 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.692555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.692657 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.692694 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 10:57:13.692811 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 10:57:13.692924 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":83,"threads_started":1}
I20260927 10:57:13.693149 32647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.688106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53270 (local address 127.14.68.3:45123)
0927 10:57:13.688510 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:13.688523 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:13.688552 (+ 29us) server_negotiation.cc:407] Connection header received
0927 10:57:13.688632 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:13.688637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:13.688868 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:13.689023 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:13.689351 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.690187 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.691586 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.692216 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.692317 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.692491 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:13.692873 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 10:57:13.692975 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":95,"threads_started":1}
I20260927 10:57:13.693663 32645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.687833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46944 (local address 127.14.68.1:33223)
0927 10:57:13.688810 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:13.688819 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:13.688838 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:57:13.688895 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:13.688898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:13.689089 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:13.689237 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:13.689461 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.690639 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.692285 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.692819 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.692922 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.693087 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:13.693406 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 10:57:13.693510 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":890,"thread_start_us":115,"threads_started":1}
I20260927 10:57:13.694214 32648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.688447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.14.68.2:44597)
0927 10:57:13.688822 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:13.688833 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:13.688854 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:57:13.688928 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:13.688936 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:13.689107 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:13.689291 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:13.689630 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.690720 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.692161 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.692948 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.693041 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.693900 (+ 859us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:13.693954 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 10:57:13.694055 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":108,"threads_started":1}
I20260927 10:57:13.694271 32643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.687295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:33223 (local address 127.0.0.1:46944)
0927 10:57:13.687645 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:13.687729 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:13.687854 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:13.689241 (+ 1387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:13.689244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:13.689263 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:13.689358 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.689364 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.690778 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.690784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.692163 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.692171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.693152 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.693156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.693253 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.693275 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:57:13.693355 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 10:57:13.694146 (+ 791us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":205,"thread_start_us":82,"threads_started":1}
I20260927 10:57:13.694567 32646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.688026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:44597 (local address 127.0.0.1:36526)
0927 10:57:13.688496 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:13.688507 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:13.688585 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:13.689407 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:13.689410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:13.689427 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:13.689504 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.689509 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.690830 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.690832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.692030 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.692040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.693529 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.693533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.693627 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.693650 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:57:13.693731 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 10:57:13.694446 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":2,"thread_start_us":91,"threads_started":1}
I20260927 10:57:13.696305 32566 tablet_service.cc:1467] Processing CreateTablet for tablet 247745bffbeb40b99d7119593466f103 (DEFAULT_TABLE table=test-table [id=14d91079c78343959e660f1924d367b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:13.696650 32304 tablet_service.cc:1467] Processing CreateTablet for tablet 247745bffbeb40b99d7119593466f103 (DEFAULT_TABLE table=test-table [id=14d91079c78343959e660f1924d367b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:13.697395 32566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 247745bffbeb40b99d7119593466f103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:13.697080 32435 tablet_service.cc:1467] Processing CreateTablet for tablet 247745bffbeb40b99d7119593466f103 (DEFAULT_TABLE table=test-table [id=14d91079c78343959e660f1924d367b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:13.697574 32304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 247745bffbeb40b99d7119593466f103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:13.697965 32435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 247745bffbeb40b99d7119593466f103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:13.703248 32649 tablet_bootstrap.cc:492] T 247745bffbeb40b99d7119593466f103 P a6a2101b239246bcb948bb20853b507a: Bootstrap starting.
I20260927 10:57:13.704465 32651 tablet_bootstrap.cc:492] T 247745bffbeb40b99d7119593466f103 P ec2622a251784b229c8d177b5a756f47: Bootstrap starting.
I20260927 10:57:13.704787 32650 tablet_bootstrap.cc:492] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a: Bootstrap starting.
I20260927 10:57:13.705883 32649 tablet_bootstrap.cc:654] T 247745bffbeb40b99d7119593466f103 P a6a2101b239246bcb948bb20853b507a: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:13.706554 32650 tablet_bootstrap.cc:654] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:13.706736 32649 log.cc:824] T 247745bffbeb40b99d7119593466f103 P a6a2101b239246bcb948bb20853b507a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:13.706987 32651 tablet_bootstrap.cc:654] T 247745bffbeb40b99d7119593466f103 P ec2622a251784b229c8d177b5a756f47: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:13.707343 32650 log.cc:824] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:13.707700 32651 log.cc:824] T 247745bffbeb40b99d7119593466f103 P ec2622a251784b229c8d177b5a756f47: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:13.708679 32649 tablet_bootstrap.cc:492] T 247745bffbeb40b99d7119593466f103 P a6a2101b239246bcb948bb20853b507a: No bootstrap required, opened a new log
I20260927 10:57:13.708869 32649 ts_tablet_manager.cc:1397] T 247745bffbeb40b99d7119593466f103 P a6a2101b239246bcb948bb20853b507a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:57:13.708995 32650 tablet_bootstrap.cc:492] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a: No bootstrap required, opened a new log
I20260927 10:57:13.709180 32650 ts_tablet_manager.cc:1397] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:57:13.712355 32649 raft_consensus.cc:348] T 247745bffbeb40b99d7119593466f103 P a6a2101b239246bcb948bb20853b507a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a2101b239246bcb948bb20853b507a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33223 } } peers { permanent_uuid: "af32407b8e90478f9c1203322448e74a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45123 } } peers { permanent_uuid: "ec2622a251784b229c8d177b5a756f47" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 44597 } }
I20260927 10:57:13.712602 32649 raft_consensus.cc:374] T 247745bffbeb40b99d7119593466f103 P a6a2101b239246bcb948bb20853b507a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:13.712654 32649 raft_consensus.cc:729] T 247745bffbeb40b99d7119593466f103 P a6a2101b239246bcb948bb20853b507a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a2101b239246bcb948bb20853b507a, State: Initialized, Role: FOLLOWER
I20260927 10:57:13.712759 32650 raft_consensus.cc:348] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a2101b239246bcb948bb20853b507a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33223 } } peers { permanent_uuid: "af32407b8e90478f9c1203322448e74a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45123 } } peers { permanent_uuid: "ec2622a251784b229c8d177b5a756f47" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 44597 } }
I20260927 10:57:13.713008 32650 raft_consensus.cc:374] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:13.713063 32650 raft_consensus.cc:729] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af32407b8e90478f9c1203322448e74a, State: Initialized, Role: FOLLOWER
I20260927 10:57:13.713095 32649 consensus_queue.cc:260] T 247745bffbeb40b99d7119593466f103 P a6a2101b239246bcb948bb20853b507a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a2101b239246bcb948bb20853b507a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33223 } } peers { permanent_uuid: "af32407b8e90478f9c1203322448e74a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45123 } } peers { permanent_uuid: "ec2622a251784b229c8d177b5a756f47" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 44597 } }
I20260927 10:57:13.713552 32650 consensus_queue.cc:260] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a2101b239246bcb948bb20853b507a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33223 } } peers { permanent_uuid: "af32407b8e90478f9c1203322448e74a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45123 } } peers { permanent_uuid: "ec2622a251784b229c8d177b5a756f47" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 44597 } }
I20260927 10:57:13.713762 32651 tablet_bootstrap.cc:492] T 247745bffbeb40b99d7119593466f103 P ec2622a251784b229c8d177b5a756f47: No bootstrap required, opened a new log
I20260927 10:57:13.713940 32651 ts_tablet_manager.cc:1397] T 247745bffbeb40b99d7119593466f103 P ec2622a251784b229c8d177b5a756f47: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20260927 10:57:13.714159 32649 ts_tablet_manager.cc:1428] T 247745bffbeb40b99d7119593466f103 P a6a2101b239246bcb948bb20853b507a: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260927 10:57:13.714221 32632 heartbeater.cc:503] Master 127.14.68.62:41241 was elected leader, sending a full tablet report...
I20260927 10:57:13.714452 32650 ts_tablet_manager.cc:1428] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260927 10:57:13.714469 32370 heartbeater.cc:503] Master 127.14.68.62:41241 was elected leader, sending a full tablet report...
I20260927 10:57:13.717064 32651 raft_consensus.cc:348] T 247745bffbeb40b99d7119593466f103 P ec2622a251784b229c8d177b5a756f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a2101b239246bcb948bb20853b507a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33223 } } peers { permanent_uuid: "af32407b8e90478f9c1203322448e74a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45123 } } peers { permanent_uuid: "ec2622a251784b229c8d177b5a756f47" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 44597 } }
I20260927 10:57:13.717295 32651 raft_consensus.cc:374] T 247745bffbeb40b99d7119593466f103 P ec2622a251784b229c8d177b5a756f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:13.717357 32651 raft_consensus.cc:729] T 247745bffbeb40b99d7119593466f103 P ec2622a251784b229c8d177b5a756f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec2622a251784b229c8d177b5a756f47, State: Initialized, Role: FOLLOWER
I20260927 10:57:13.717772 32651 consensus_queue.cc:260] T 247745bffbeb40b99d7119593466f103 P ec2622a251784b229c8d177b5a756f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a2101b239246bcb948bb20853b507a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33223 } } peers { permanent_uuid: "af32407b8e90478f9c1203322448e74a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45123 } } peers { permanent_uuid: "ec2622a251784b229c8d177b5a756f47" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 44597 } }
I20260927 10:57:13.718506 32501 heartbeater.cc:503] Master 127.14.68.62:41241 was elected leader, sending a full tablet report...
I20260927 10:57:13.718678 32651 ts_tablet_manager.cc:1428] T 247745bffbeb40b99d7119593466f103 P ec2622a251784b229c8d177b5a756f47: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
W20260927 10:57:13.720778 32502 tablet.cc:2369] T 247745bffbeb40b99d7119593466f103 P ec2622a251784b229c8d177b5a756f47: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:57:13.787117 32371 tablet.cc:2369] T 247745bffbeb40b99d7119593466f103 P a6a2101b239246bcb948bb20853b507a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:57:13.896052 32633 tablet.cc:2369] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:57:13.950582 32656 raft_consensus.cc:482] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:57:13.950904 32656 raft_consensus.cc:504] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a2101b239246bcb948bb20853b507a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33223 } } peers { permanent_uuid: "af32407b8e90478f9c1203322448e74a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45123 } } peers { permanent_uuid: "ec2622a251784b229c8d177b5a756f47" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 44597 } }
I20260927 10:57:13.952209 32656 leader_election.cc:288] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6a2101b239246bcb948bb20853b507a (127.14.68.1:33223), ec2622a251784b229c8d177b5a756f47 (127.14.68.2:44597)
I20260927 10:57:13.956154 32634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.952463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:44597 (local address 127.14.68.3:35999)
0927 10:57:13.952596 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:13.952614 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:13.952726 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:13.953038 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:13.953041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:13.953062 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:13.953217 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.953225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.954096 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.954099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.954982 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.954992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.955727 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.955734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.955836 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.955858 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:57:13.955934 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 10:57:13.955988 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":27}
I20260927 10:57:13.956187 32648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.952567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:35999 (local address 127.14.68.2:44597)
0927 10:57:13.952683 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:13.952688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:13.952704 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:57:13.952857 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:13.952862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:13.952920 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:13.952998 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:13.953360 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.953982 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.955105 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.955566 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.955654 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.955771 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:13.955970 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 10:57:13.956056 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 10:57:13.956225 32658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.952466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:33223 (local address 127.14.68.3:50219)
0927 10:57:13.953033 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:13.953046 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:13.953143 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:13.953395 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:13.953397 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:13.953414 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:13.953535 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.953541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.954303 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.954306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.955122 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.955132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.955803 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.955807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.955907 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.955955 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 10:57:13.956032 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 10:57:13.956091 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":268,"thread_start_us":100,"threads_started":1}
I20260927 10:57:13.956256 32645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.952546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:50219 (local address 127.14.68.1:33223)
0927 10:57:13.952690 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:13.952700 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:13.953138 (+ 438us) server_negotiation.cc:407] Connection header received
0927 10:57:13.953222 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:13.953225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:13.953285 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:13.953364 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:13.953652 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.954192 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.955242 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.955670 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.955791 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.955850 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:13.956063 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 10:57:13.956135 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:57:13.956856 32455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247745bffbeb40b99d7119593466f103" candidate_uuid: "af32407b8e90478f9c1203322448e74a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec2622a251784b229c8d177b5a756f47" is_pre_election: true
I20260927 10:57:13.957136 32324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247745bffbeb40b99d7119593466f103" candidate_uuid: "af32407b8e90478f9c1203322448e74a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a2101b239246bcb948bb20853b507a" is_pre_election: true
I20260927 10:57:13.957228 32455 raft_consensus.cc:2454] T 247745bffbeb40b99d7119593466f103 P ec2622a251784b229c8d177b5a756f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af32407b8e90478f9c1203322448e74a in term 0.
I20260927 10:57:13.957424 32324 raft_consensus.cc:2454] T 247745bffbeb40b99d7119593466f103 P a6a2101b239246bcb948bb20853b507a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af32407b8e90478f9c1203322448e74a in term 0.
I20260927 10:57:13.957736 32520 leader_election.cc:302] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af32407b8e90478f9c1203322448e74a, ec2622a251784b229c8d177b5a756f47; no voters:
I20260927 10:57:13.958001 32656 raft_consensus.cc:2781] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:57:13.958079 32656 raft_consensus.cc:482] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:57:13.958120 32656 raft_consensus.cc:3037] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:13.959088 32656 raft_consensus.cc:504] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a2101b239246bcb948bb20853b507a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33223 } } peers { permanent_uuid: "af32407b8e90478f9c1203322448e74a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45123 } } peers { permanent_uuid: "ec2622a251784b229c8d177b5a756f47" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 44597 } }
I20260927 10:57:13.959398 32656 leader_election.cc:288] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a [CANDIDATE]: Term 1 election: Requested vote from peers a6a2101b239246bcb948bb20853b507a (127.14.68.1:33223), ec2622a251784b229c8d177b5a756f47 (127.14.68.2:44597)
I20260927 10:57:13.959728 32324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247745bffbeb40b99d7119593466f103" candidate_uuid: "af32407b8e90478f9c1203322448e74a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a2101b239246bcb948bb20853b507a"
I20260927 10:57:13.959709 32455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247745bffbeb40b99d7119593466f103" candidate_uuid: "af32407b8e90478f9c1203322448e74a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec2622a251784b229c8d177b5a756f47"
I20260927 10:57:13.959864 32455 raft_consensus.cc:3037] T 247745bffbeb40b99d7119593466f103 P ec2622a251784b229c8d177b5a756f47 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:13.959834 32324 raft_consensus.cc:3037] T 247745bffbeb40b99d7119593466f103 P a6a2101b239246bcb948bb20853b507a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:13.960830 32324 raft_consensus.cc:2454] T 247745bffbeb40b99d7119593466f103 P a6a2101b239246bcb948bb20853b507a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af32407b8e90478f9c1203322448e74a in term 1.
I20260927 10:57:13.960933 32455 raft_consensus.cc:2454] T 247745bffbeb40b99d7119593466f103 P ec2622a251784b229c8d177b5a756f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af32407b8e90478f9c1203322448e74a in term 1.
I20260927 10:57:13.961175 32521 leader_election.cc:302] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6a2101b239246bcb948bb20853b507a, af32407b8e90478f9c1203322448e74a; no voters:
I20260927 10:57:13.961352 32656 raft_consensus.cc:2781] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:13.961561 32656 raft_consensus.cc:686] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a [term 1 LEADER]: Becoming Leader. State: Replica: af32407b8e90478f9c1203322448e74a, State: Running, Role: LEADER
I20260927 10:57:13.961915 32656 consensus_queue.cc:237] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a2101b239246bcb948bb20853b507a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33223 } } peers { permanent_uuid: "af32407b8e90478f9c1203322448e74a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45123 } } peers { permanent_uuid: "ec2622a251784b229c8d177b5a756f47" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 44597 } }
I20260927 10:57:13.964938 32182 catalog_manager.cc:5495] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a reported cstate change: term changed from 0 to 1, leader changed from <none> to af32407b8e90478f9c1203322448e74a (127.14.68.3). New cstate: current_term: 1 leader_uuid: "af32407b8e90478f9c1203322448e74a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a2101b239246bcb948bb20853b507a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af32407b8e90478f9c1203322448e74a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 45123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec2622a251784b229c8d177b5a756f47" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 44597 } health_report { overall_health: UNKNOWN } } }
I20260927 10:57:13.984171 32647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.981160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53278 (local address 127.14.68.3:45123)
0927 10:57:13.981278 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:13.981282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:13.981294 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:57:13.981355 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:13.981358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:13.981406 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:13.981488 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:57:13.981691 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.982174 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.982939 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.983131 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.983215 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.983452 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:57:13.983859 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:57:13.983946 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:13.984002 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 10:57:13.984043 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:57:13.989708 32455 raft_consensus.cc:1261] T 247745bffbeb40b99d7119593466f103 P ec2622a251784b229c8d177b5a756f47 [term 1 FOLLOWER]: Refusing update from remote peer af32407b8e90478f9c1203322448e74a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:13.989709 32324 raft_consensus.cc:1261] T 247745bffbeb40b99d7119593466f103 P a6a2101b239246bcb948bb20853b507a [term 1 FOLLOWER]: Refusing update from remote peer af32407b8e90478f9c1203322448e74a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:13.990411 32656 consensus_queue.cc:1035] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec2622a251784b229c8d177b5a756f47" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 44597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:13.990689 32659 consensus_queue.cc:1035] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a2101b239246bcb948bb20853b507a" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:13.997577 32662 mvcc.cc:205] Tried to move back new op lower bound from 7333915172815233024 to 7333915172710526976. Current Snapshot: MvccSnapshot[applied={T|T < 7333915172815233024}]
I20260927 10:57:13.999293 32664 mvcc.cc:205] Tried to move back new op lower bound from 7333915172815233024 to 7333915172710526976. Current Snapshot: MvccSnapshot[applied={T|T < 7333915172815233024}]
I20260927 10:57:14.007118 32182 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:57:14.007292 32182 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:57:14.009828 32182 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15fc7b6ab71e468faa8227fb15e6e228: Sending DeleteTablet for 3 replicas of tablet 247745bffbeb40b99d7119593466f103
I20260927 10:57:14.010577 32304 tablet_service.cc:1514] Processing DeleteTablet for tablet 247745bffbeb40b99d7119593466f103 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:14 UTC) from {username='slave'} at 127.0.0.1:46944
I20260927 10:57:14.010684 32435 tablet_service.cc:1514] Processing DeleteTablet for tablet 247745bffbeb40b99d7119593466f103 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:14 UTC) from {username='slave'} at 127.0.0.1:36526
I20260927 10:57:14.010876 32566 tablet_service.cc:1514] Processing DeleteTablet for tablet 247745bffbeb40b99d7119593466f103 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:14 UTC) from {username='slave'} at 127.0.0.1:53270
I20260927 10:57:14.011385 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 32242
I20260927 10:57:14.011278 32672 tablet_replica.cc:330] T 247745bffbeb40b99d7119593466f103 P a6a2101b239246bcb948bb20853b507a: stopping tablet replica
I20260927 10:57:14.012292 32673 tablet_replica.cc:330] T 247745bffbeb40b99d7119593466f103 P ec2622a251784b229c8d177b5a756f47: stopping tablet replica
I20260927 10:57:14.012609 32673 raft_consensus.cc:2229] T 247745bffbeb40b99d7119593466f103 P ec2622a251784b229c8d177b5a756f47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:57:14.012955 32673 raft_consensus.cc:2258] T 247745bffbeb40b99d7119593466f103 P ec2622a251784b229c8d177b5a756f47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:14.013134 32674 tablet_replica.cc:330] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a: stopping tablet replica
I20260927 10:57:14.013422 32674 raft_consensus.cc:2229] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:57:14.013769 32674 raft_consensus.cc:2258] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:14.014386 32673 ts_tablet_manager.cc:1905] T 247745bffbeb40b99d7119593466f103 P ec2622a251784b229c8d177b5a756f47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:57:14.015370 32674 ts_tablet_manager.cc:1905] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:57:14.017776 32673 ts_tablet_manager.cc:1918] T 247745bffbeb40b99d7119593466f103 P ec2622a251784b229c8d177b5a756f47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:57:14.017875 32673 log.cc:1196] T 247745bffbeb40b99d7119593466f103 P ec2622a251784b229c8d177b5a756f47: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/247745bffbeb40b99d7119593466f103
I20260927 10:57:14.018193 32673 ts_tablet_manager.cc:1939] T 247745bffbeb40b99d7119593466f103 P ec2622a251784b229c8d177b5a756f47: Deleting consensus metadata
I20260927 10:57:14.018551 32674 ts_tablet_manager.cc:1918] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:57:14.018712 32674 log.cc:1196] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/247745bffbeb40b99d7119593466f103
I20260927 10:57:14.019130 32674 ts_tablet_manager.cc:1939] T 247745bffbeb40b99d7119593466f103 P af32407b8e90478f9c1203322448e74a: Deleting consensus metadata
I20260927 10:57:14.019539 32167 catalog_manager.cc:4841] TS ec2622a251784b229c8d177b5a756f47 (127.14.68.2:44597): tablet 247745bffbeb40b99d7119593466f103 (table test-table [id=14d91079c78343959e660f1924d367b7]) successfully deleted
W20260927 10:57:14.019682 32168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.68.1:33223 (error 108)
I20260927 10:57:14.020567 32167 catalog_manager.cc:4841] TS af32407b8e90478f9c1203322448e74a (127.14.68.3:45123): tablet 247745bffbeb40b99d7119593466f103 (table test-table [id=14d91079c78343959e660f1924d367b7]) successfully deleted
I20260927 10:57:14.020730 32646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:14.020402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:33223 (local address 127.0.0.1:46952)
0927 10:57:14.020569 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:14.020624 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.1:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260927 10:57:14.020895 32168 catalog_manager.cc:4584] TS a6a2101b239246bcb948bb20853b507a (127.14.68.1:33223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 247745bffbeb40b99d7119593466f103: Network error: Client connection negotiation failed: client connection to 127.14.68.1:33223: connect: Connection refused (error 111)
I20260927 10:57:14.021317 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 32373
I20260927 10:57:14.027069 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 32504
I20260927 10:57:14.032985 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 32151
2026-09-27T10:57:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1179 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:14 dist-test-slave-tz51 krb5kdc[32678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:14 dist-test-slave-tz51 krb5kdc[32678](info): set up 2 sockets
Sep 27 10:57:14 dist-test-slave-tz51 krb5kdc[32678](info): commencing operation
krb5kdc: starting...
W20260927 10:57:15.079530 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:57:15 dist-test-slave-tz51 krb5kdc[32678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:57:15.203074 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:44971
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:44971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:15.309401 32691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:15.309636 32691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:15.309680 32691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:15.313491 32691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:15.313534 32691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:15.313551 32691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:15.313567 32691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:15.313582 32691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:15.318571 32691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:44971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:44971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:15.319442 32691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:15.320353 32691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:15.328688 32699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:15.328701 32697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:15.328706 32696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:15.329080 32691 server_base.cc:1034] running on GCE node
I20260927 10:57:15.329584 32691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:15.330610 32691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:15.331776 32691 hybrid_clock.cc:648] HybridClock initialized: now 1790506635331759 us; error 24 us; skew 500 ppm
Sep 27 10:57:15 dist-test-slave-tz51 krb5kdc[32678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506635, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:15.334443 32691 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:57:15.335274 32691 webserver.cc:460] Webserver started at http://127.14.68.62:33241/ using document root <none> and password file <none>
I20260927 10:57:15.335850 32691 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:15.335937 32691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:15.336154 32691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:15.338114 32691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "610bcbbea88542f5bfac8e0bc175d005"
format_stamp: "Formatted at 2026-09-27 10:57:15 on dist-test-slave-tz51"
server_key: "7d3acdd8e5b1949bb4ca2247af7922da"
server_key_iv: "974851562ed1d47b6be24b5a093562d9"
server_key_version: "encryptionkey@0"
I20260927 10:57:15.338647 32691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "610bcbbea88542f5bfac8e0bc175d005"
format_stamp: "Formatted at 2026-09-27 10:57:15 on dist-test-slave-tz51"
server_key: "7d3acdd8e5b1949bb4ca2247af7922da"
server_key_iv: "974851562ed1d47b6be24b5a093562d9"
server_key_version: "encryptionkey@0"
I20260927 10:57:15.342133 32691 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:57:15.344309 32706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:15.345392 32691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:15.345522 32691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "610bcbbea88542f5bfac8e0bc175d005"
format_stamp: "Formatted at 2026-09-27 10:57:15 on dist-test-slave-tz51"
server_key: "7d3acdd8e5b1949bb4ca2247af7922da"
server_key_iv: "974851562ed1d47b6be24b5a093562d9"
server_key_version: "encryptionkey@0"
I20260927 10:57:15.345628 32691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:15.360167 32691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:15.363160 32691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:15.363343 32691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:15.370048 32691 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:44971
I20260927 10:57:15.370059 32758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:44971 every 8 connection(s)
I20260927 10:57:15.370916 32691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:15.373548 32759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:15.378468 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 32691
I20260927 10:57:15.378626 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:15.378608 32759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005: Bootstrap starting.
I20260927 10:57:15.378890 14608 external_mini_cluster.cc:1428] Setting key 5710e7f2cf9bbeb19ee0086d855308f0
I20260927 10:57:15.380970 32759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:15.381690 32759 log.cc:824] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:15.383457 32759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005: No bootstrap required, opened a new log
Sep 27 10:57:15 dist-test-slave-tz51 krb5kdc[32678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:57:15.387104 32759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610bcbbea88542f5bfac8e0bc175d005" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44971 } }
I20260927 10:57:15.387362 32759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:15.387421 32759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 610bcbbea88542f5bfac8e0bc175d005, State: Initialized, Role: FOLLOWER
I20260927 10:57:15.387883 32759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610bcbbea88542f5bfac8e0bc175d005" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44971 } }
I20260927 10:57:15.388001 32759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:15.388100 32759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:15.388206 32759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:15.389130 32759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610bcbbea88542f5bfac8e0bc175d005" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44971 } }
I20260927 10:57:15.389467 32759 leader_election.cc:302] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 610bcbbea88542f5bfac8e0bc175d005; no voters:
I20260927 10:57:15.389816 32759 leader_election.cc:288] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:15.389870 32764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:15.390039 32764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005 [term 1 LEADER]: Becoming Leader. State: Replica: 610bcbbea88542f5bfac8e0bc175d005, State: Running, Role: LEADER
I20260927 10:57:15.390311 32764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610bcbbea88542f5bfac8e0bc175d005" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44971 } }
I20260927 10:57:15.390944 32759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:15.391985 32766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 610bcbbea88542f5bfac8e0bc175d005. Latest consensus state: current_term: 1 leader_uuid: "610bcbbea88542f5bfac8e0bc175d005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610bcbbea88542f5bfac8e0bc175d005" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44971 } } }
I20260927 10:57:15.392020 32765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "610bcbbea88542f5bfac8e0bc175d005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610bcbbea88542f5bfac8e0bc175d005" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44971 } } }
I20260927 10:57:15.392654 32765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:15.392908 32766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:15.393579 32762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:15.380088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59500 (local address 127.14.68.62:44971)
0927 10:57:15.380456 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:15.380472 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:15.380504 (+ 32us) server_negotiation.cc:407] Connection header received
0927 10:57:15.381123 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:15.381152 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:15.381452 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:15.381714 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:15.382560 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:15.383483 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:15.384430 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:15.384893 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:15.385059 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:15.387741 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:15.387747 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:15.387769 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:15.387792 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:15.389522 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:15.390056 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:15.390063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:15.390071 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:15.390215 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:15.390525 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:15.390528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:15.390529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:15.390959 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:15.391163 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:15.392360 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0927 10:57:15.392682 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":123,"threads_started":1}
W20260927 10:57:15.395637 312 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:57:15.395751 312 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:57:15.395925 309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:15.396745 309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:15.401530 309 catalog_manager.cc:1347] Generated new cluster ID: df7411faace04ad3966774718e63a0f6
I20260927 10:57:15.401587 309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:15.408917 309 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:15.409734 309 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:15.415908 309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005: Generated new TSK 0
I20260927 10:57:15.416522 309 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
I20260927 10:57:15.451031 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:44971
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:15.559317 317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:15.559556 317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:15.559595 317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:15.563572 317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:15.563628 317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:15.563735 317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:57:15.568917 317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:44971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:15.569895 317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:15.570881 317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:15.580310 322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:15.580327 323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:15.580331 325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:15.580608 317 server_base.cc:1034] running on GCE node
I20260927 10:57:15.580951 317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:15.581568 317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:15.582726 317 hybrid_clock.cc:648] HybridClock initialized: now 1790506635582713 us; error 27 us; skew 500 ppm
Sep 27 10:57:15 dist-test-slave-tz51 krb5kdc[32678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506635, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:15.585518 317 init.cc:377] Logged in from keytab as kudu/127.14.68.1@KRBTEST.COM (short username kudu)
I20260927 10:57:15.586333 317 webserver.cc:460] Webserver started at http://127.14.68.1:39239/ using document root <none> and password file <none>
I20260927 10:57:15.586836 317 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:15.586890 317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:15.587047 317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:15.589037 317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "114410c24864480dabd05dc81d0fda31"
format_stamp: "Formatted at 2026-09-27 10:57:15 on dist-test-slave-tz51"
server_key: "7a04cddedb93e75c2202a71fa9f18db6"
server_key_iv: "16a2fdf9db22c35bed36883f4964d99d"
server_key_version: "encryptionkey@0"
I20260927 10:57:15.589537 317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "114410c24864480dabd05dc81d0fda31"
format_stamp: "Formatted at 2026-09-27 10:57:15 on dist-test-slave-tz51"
server_key: "7a04cddedb93e75c2202a71fa9f18db6"
server_key_iv: "16a2fdf9db22c35bed36883f4964d99d"
server_key_version: "encryptionkey@0"
I20260927 10:57:15.593118 317 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:57:15.595340 332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:15.596552 317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:15.596666 317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "114410c24864480dabd05dc81d0fda31"
format_stamp: "Formatted at 2026-09-27 10:57:15 on dist-test-slave-tz51"
server_key: "7a04cddedb93e75c2202a71fa9f18db6"
server_key_iv: "16a2fdf9db22c35bed36883f4964d99d"
server_key_version: "encryptionkey@0"
I20260927 10:57:15.596766 317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:15.609454 317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:15.612586 317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:15.612776 317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:15.613353 317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:15.614145 317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:15.614192 317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:15.614255 317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:15.614284 317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:15.623049 317 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:41161
I20260927 10:57:15.623070 445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:41161 every 8 connection(s)
I20260927 10:57:15.623915 317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:57:15.626900 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 317
I20260927 10:57:15.627054 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:57:15.627427 14608 external_mini_cluster.cc:1428] Setting key 502ee7f4f1b9cd7608288d3583dba79c
WARNING: no policy specified for kudu/127.14.68.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.2@KRBTEST.COM" created.
Sep 27 10:57:15 dist-test-slave-tz51 krb5kdc[32678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506635, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:57:15.643581 32762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:15.625606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:40851 (local address 127.14.68.62:44971)
0927 10:57:15.625747 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:15.625752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:15.626611 (+ 859us) server_negotiation.cc:407] Connection header received
0927 10:57:15.628362 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:15.628365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:15.628409 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:15.628492 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:15.630000 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:15.630588 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:15.631723 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:15.631989 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:15.632102 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:15.639884 (+ 7782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:15.639887 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:15.639889 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:15.639914 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:15.641570 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:15.642072 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:15.642076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:15.642078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:15.642160 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:15.642496 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:15.642501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:15.642502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:15.642690 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:15.642791 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:15.643283 (+ 492us) server_negotiation.cc:299] Negotiation successful
0927 10:57:15.643402 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:57:15.644392 448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:15.626039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44971 (local address 127.14.68.1:40851)
0927 10:57:15.626468 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:15.626509 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:15.628140 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:15.628638 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:15.628645 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:15.629170 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:15.629846 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:15.629867 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:15.630710 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:15.630714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:15.631582 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:15.631591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:15.632136 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:15.632140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:15.632309 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:15.636976 (+ 4667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:15.636992 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:15.639660 (+ 2668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:15.641680 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:15.641686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:15.641701 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:15.641952 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:15.642256 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:15.642259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:15.642261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:15.642403 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:15.642801 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:15.642809 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:15.643057 (+ 248us) client_negotiation.cc:769] Sending connection context
0927 10:57:15.643244 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 10:57:15.643553 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":114,"threads_started":1}
I20260927 10:57:15.645833 446 heartbeater.cc:347] Connected to a master server at 127.14.68.62:44971
I20260927 10:57:15.646091 446 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:15.646566 446 heartbeater.cc:511] Master 127.14.68.62:44971 requested a full tablet report, sending...
I20260927 10:57:15.648109 32723 ts_manager.cc:194] Registered new tserver with Master: 114410c24864480dabd05dc81d0fda31 (127.14.68.1:41161)
I20260927 10:57:15.649232 32723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:40851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
I20260927 10:57:15.654562 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:44971
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:15.765825 451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:15.766069 451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:15.766111 451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:15.770152 451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:15.770195 451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:15.770280 451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:57:15.775151 451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:44971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:15.776088 451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:15.777197 451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:15.786316 457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:15.786367 456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:15.786322 459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:15.786478 451 server_base.cc:1034] running on GCE node
I20260927 10:57:15.786896 451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:15.787463 451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:15.788615 451 hybrid_clock.cc:648] HybridClock initialized: now 1790506635788593 us; error 34 us; skew 500 ppm
Sep 27 10:57:15 dist-test-slave-tz51 krb5kdc[32678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506635, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:15.791263 451 init.cc:377] Logged in from keytab as kudu/127.14.68.2@KRBTEST.COM (short username kudu)
I20260927 10:57:15.792538 451 webserver.cc:460] Webserver started at http://127.14.68.2:45687/ using document root <none> and password file <none>
I20260927 10:57:15.793102 451 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:15.793188 451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:15.793390 451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:15.795252 451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "5db0920a85d64ed1a68aafcc90dfd515"
format_stamp: "Formatted at 2026-09-27 10:57:15 on dist-test-slave-tz51"
server_key: "200ad21c07ece2eba7edba467aa689f1"
server_key_iv: "feac1c352ce14e82dccef9434ce2e244"
server_key_version: "encryptionkey@0"
I20260927 10:57:15.795879 451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "5db0920a85d64ed1a68aafcc90dfd515"
format_stamp: "Formatted at 2026-09-27 10:57:15 on dist-test-slave-tz51"
server_key: "200ad21c07ece2eba7edba467aa689f1"
server_key_iv: "feac1c352ce14e82dccef9434ce2e244"
server_key_version: "encryptionkey@0"
I20260927 10:57:15.799275 451 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:57:15.801591 466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:15.802711 451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:15.802845 451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "5db0920a85d64ed1a68aafcc90dfd515"
format_stamp: "Formatted at 2026-09-27 10:57:15 on dist-test-slave-tz51"
server_key: "200ad21c07ece2eba7edba467aa689f1"
server_key_iv: "feac1c352ce14e82dccef9434ce2e244"
server_key_version: "encryptionkey@0"
I20260927 10:57:15.802944 451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:15.824600 451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:15.827523 451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:15.827788 451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:15.828399 451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:15.829209 451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:15.829277 451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:15.829348 451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:15.829389 451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:15.838090 451 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:39125
I20260927 10:57:15.838117 579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:39125 every 8 connection(s)
I20260927 10:57:15.838954 451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:57:15.840615 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 451
I20260927 10:57:15.840781 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:57:15.841110 14608 external_mini_cluster.cc:1428] Setting key 0a20f8362dc6c8c18dc7906c508ca3db
Sep 27 10:57:15 dist-test-slave-tz51 krb5kdc[32678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506635, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.68.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.3@KRBTEST.COM" created.
I20260927 10:57:15.853062 32762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:15.840769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:53223 (local address 127.14.68.62:44971)
0927 10:57:15.840926 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:15.840931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:15.841841 (+ 910us) server_negotiation.cc:407] Connection header received
0927 10:57:15.843220 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:15.843224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:15.843277 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:15.843358 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:15.844647 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:15.845179 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:15.846415 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:15.846643 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:15.846754 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:15.849419 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:15.849424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:15.849429 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:15.849459 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:15.850957 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:15.851426 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:15.851429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:15.851430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:15.851506 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:15.851839 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:15.851841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:15.851842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:15.852057 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:15.852176 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:15.852657 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 10:57:15.852801 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:57:15.853873 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:15.841234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44971 (local address 127.14.68.2:53223)
0927 10:57:15.841700 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:15.841750 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:15.843011 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:15.843529 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:15.843536 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:15.844054 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:15.844507 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:15.844520 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:15.845304 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:15.845307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:15.846293 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:15.846302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:15.846790 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:15.846794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:15.846959 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:15.847605 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:15.847621 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:15.849255 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:15.851059 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:15.851064 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:15.851076 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:15.851343 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:15.851582 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:15.851584 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:15.851586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:15.851769 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:15.852173 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:15.852178 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:15.852399 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 10:57:15.852547 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 10:57:15.852875 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":131,"threads_started":1}
I20260927 10:57:15.855113 580 heartbeater.cc:347] Connected to a master server at 127.14.68.62:44971
I20260927 10:57:15.855363 580 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:15.855901 580 heartbeater.cc:511] Master 127.14.68.62:44971 requested a full tablet report, sending...
I20260927 10:57:15.856967 32723 ts_manager.cc:194] Registered new tserver with Master: 5db0920a85d64ed1a68aafcc90dfd515 (127.14.68.2:39125)
I20260927 10:57:15.857803 32723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:53223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
I20260927 10:57:15.868777 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:44971
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:15.974792 585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:15.975080 585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:15.975163 585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:15.979173 585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:15.979269 585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:15.979360 585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:57:15.985150 585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:44971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:15.986176 585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:15.987207 585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:15.996785 593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:15.996798 591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:15.996786 590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:15.996833 585 server_base.cc:1034] running on GCE node
I20260927 10:57:15.997383 585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:15.998014 585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:15.999239 585 hybrid_clock.cc:648] HybridClock initialized: now 1790506635999224 us; error 25 us; skew 500 ppm
Sep 27 10:57:15 dist-test-slave-tz51 krb5kdc[32678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506635, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:16.002151 585 init.cc:377] Logged in from keytab as kudu/127.14.68.3@KRBTEST.COM (short username kudu)
I20260927 10:57:16.002975 585 webserver.cc:460] Webserver started at http://127.14.68.3:37229/ using document root <none> and password file <none>
I20260927 10:57:16.003463 585 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:16.003520 585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:16.003677 585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:16.005611 585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "698d3978bfd8449e8384c65e4b94633a"
format_stamp: "Formatted at 2026-09-27 10:57:16 on dist-test-slave-tz51"
server_key: "d9a2101b65a5199e6a0f1383ef7b2490"
server_key_iv: "df63688cb714ac3bcaa3385b237f7882"
server_key_version: "encryptionkey@0"
I20260927 10:57:16.006110 585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "698d3978bfd8449e8384c65e4b94633a"
format_stamp: "Formatted at 2026-09-27 10:57:16 on dist-test-slave-tz51"
server_key: "d9a2101b65a5199e6a0f1383ef7b2490"
server_key_iv: "df63688cb714ac3bcaa3385b237f7882"
server_key_version: "encryptionkey@0"
I20260927 10:57:16.009614 585 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:57:16.011839 600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:16.012966 585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:57:16.013082 585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "698d3978bfd8449e8384c65e4b94633a"
format_stamp: "Formatted at 2026-09-27 10:57:16 on dist-test-slave-tz51"
server_key: "d9a2101b65a5199e6a0f1383ef7b2490"
server_key_iv: "df63688cb714ac3bcaa3385b237f7882"
server_key_version: "encryptionkey@0"
I20260927 10:57:16.013181 585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:16.029831 585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:16.033074 585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:16.033209 585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:16.033818 585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:16.034674 585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:16.034719 585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:16.034785 585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:16.034822 585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:16.043596 585 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:42331
I20260927 10:57:16.043619 713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:42331 every 8 connection(s)
I20260927 10:57:16.044521 585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
Sep 27 10:57:16 dist-test-slave-tz51 krb5kdc[32678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506635, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:57:16.054261 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 585
I20260927 10:57:16.054654 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:57:16.054996 14608 external_mini_cluster.cc:1428] Setting key f3883a314f8f33b4402539a9c5510eba
I20260927 10:57:16.057585 32762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.046228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:52645 (local address 127.14.68.62:44971)
0927 10:57:16.046364 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:16.046367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:16.047134 (+ 767us) server_negotiation.cc:407] Connection header received
0927 10:57:16.048561 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:16.048565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:16.048648 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:16.048746 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:16.049838 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.050351 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.051452 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.051679 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.051836 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.054075 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:16.054081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:16.054086 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:16.054114 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:16.055668 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:16.056169 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:16.056175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:16.056180 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:16.056272 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:16.056633 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:16.056636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:16.056638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:16.056850 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:16.056946 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:16.057379 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 10:57:16.057474 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:57:16.058362 716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.046574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:44971 (local address 127.14.68.3:52645)
0927 10:57:16.046995 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:16.047034 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:16.048361 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:16.048938 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:16.048944 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:16.049270 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:16.049700 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.049712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.050471 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.050474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.051326 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.051334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.051867 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.051871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.052032 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.052533 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:16.052548 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:16.053907 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:16.055832 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:16.055837 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:16.055849 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:16.056093 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:16.056408 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:16.056417 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:16.056419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:16.056542 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:16.056957 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:16.056967 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:16.057191 (+ 224us) client_negotiation.cc:769] Sending connection context
0927 10:57:16.057334 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 10:57:16.057549 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":120,"threads_started":1}
I20260927 10:57:16.059404 714 heartbeater.cc:347] Connected to a master server at 127.14.68.62:44971
I20260927 10:57:16.059577 714 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:16.060039 714 heartbeater.cc:511] Master 127.14.68.62:44971 requested a full tablet report, sending...
I20260927 10:57:16.060920 32723 ts_manager.cc:194] Registered new tserver with Master: 698d3978bfd8449e8384c65e4b94633a (127.14.68.3:42331)
I20260927 10:57:16.061468 32723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:52645
I20260927 10:57:16.068559 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:57:16.077548 32762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.070320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59510 (local address 127.14.68.62:44971)
0927 10:57:16.070454 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:16.070459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:16.070634 (+ 175us) server_negotiation.cc:407] Connection header received
0927 10:57:16.070776 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:16.070779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:16.070818 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:16.070908 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:16.071560 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.072124 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.072928 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.073166 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.073272 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.074624 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:16.074628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:16.074630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:16.074663 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:16.075846 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:16.076317 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:16.076320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:16.076322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:16.076401 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:16.076711 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:16.076714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:16.076716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:16.076901 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:16.077049 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:16.077314 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 10:57:16.077398 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:57:16.079957 32723 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:57:16.082805 32723 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:57:16.096199 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.091073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43640 (local address 127.14.68.2:39125)
0927 10:57:16.091631 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:16.091646 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:16.091682 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:57:16.091816 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:16.091822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:16.092006 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:16.092141 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:16.092621 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.093298 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.094853 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.095532 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.095659 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.095819 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:16.095957 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 10:57:16.096042 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":144,"threads_started":1}
I20260927 10:57:16.096196 726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.090512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:39125 (local address 127.0.0.1:43640)
0927 10:57:16.091232 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:16.091245 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:16.091338 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:16.092330 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:16.092333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:16.092350 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:16.092467 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.092474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.093419 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.093425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.094733 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.094743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.095652 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.095656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.095782 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.095819 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 10:57:16.095930 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 10:57:16.096038 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":26,"spinlock_wait_cycles":896,"thread_start_us":99,"threads_started":1}
I20260927 10:57:16.096679 725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.090334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:41161 (local address 127.0.0.1:52802)
0927 10:57:16.090632 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:16.090678 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:16.090787 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:16.092045 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:16.092052 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:16.092073 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:16.092262 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.092272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.093777 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.093781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.094983 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.094999 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.096274 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.096279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.096390 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.096410 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:57:16.096491 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 10:57:16.096551 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":78,"thread_start_us":72,"threads_started":1}
I20260927 10:57:16.097041 728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.090984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:42331 (local address 127.0.0.1:41122)
0927 10:57:16.091515 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:16.091527 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:16.091602 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:16.092342 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:16.092346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:16.092391 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:16.092482 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.092487 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.093840 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.093843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.095073 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.095083 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.096651 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.096655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.096762 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.096788 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 10:57:16.096856 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 10:57:16.096924 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":79,"spinlock_wait_cycles":768,"thread_start_us":94,"threads_started":1}
I20260927 10:57:16.097205 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.090812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52802 (local address 127.14.68.1:41161)
0927 10:57:16.091530 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:16.091542 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:16.091564 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:57:16.091652 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:16.091657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:16.091890 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:16.092046 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:16.092432 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.093631 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.095102 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.095968 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.096075 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.096210 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:16.096937 (+ 727us) server_negotiation.cc:299] Negotiation successful
0927 10:57:16.097027 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":153,"threads_started":1}
I20260927 10:57:16.097541 730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.091111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41122 (local address 127.14.68.3:42331)
0927 10:57:16.091874 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:16.091884 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:16.091903 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:57:16.092032 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:16.092037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:16.092211 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:16.092350 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:16.092600 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.093680 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.095206 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.095983 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.096075 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.096210 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:16.097295 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0927 10:57:16.097393 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":131,"threads_started":1}
I20260927 10:57:16.099002 514 tablet_service.cc:1467] Processing CreateTablet for tablet 0d22b74aaf404b70b33293e12942fc90 (DEFAULT_TABLE table=test-table [id=1d504072a318402f9957a898bc2e9438]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:16.099339 380 tablet_service.cc:1467] Processing CreateTablet for tablet 0d22b74aaf404b70b33293e12942fc90 (DEFAULT_TABLE table=test-table [id=1d504072a318402f9957a898bc2e9438]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:16.100296 380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d22b74aaf404b70b33293e12942fc90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:16.099985 514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d22b74aaf404b70b33293e12942fc90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:16.102367 648 tablet_service.cc:1467] Processing CreateTablet for tablet 0d22b74aaf404b70b33293e12942fc90 (DEFAULT_TABLE table=test-table [id=1d504072a318402f9957a898bc2e9438]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:16.103274 648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d22b74aaf404b70b33293e12942fc90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:16.107328 732 tablet_bootstrap.cc:492] T 0d22b74aaf404b70b33293e12942fc90 P 5db0920a85d64ed1a68aafcc90dfd515: Bootstrap starting.
I20260927 10:57:16.107553 731 tablet_bootstrap.cc:492] T 0d22b74aaf404b70b33293e12942fc90 P 114410c24864480dabd05dc81d0fda31: Bootstrap starting.
I20260927 10:57:16.109687 731 tablet_bootstrap.cc:654] T 0d22b74aaf404b70b33293e12942fc90 P 114410c24864480dabd05dc81d0fda31: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:16.109684 732 tablet_bootstrap.cc:654] T 0d22b74aaf404b70b33293e12942fc90 P 5db0920a85d64ed1a68aafcc90dfd515: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:16.109673 733 tablet_bootstrap.cc:492] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a: Bootstrap starting.
I20260927 10:57:16.111054 732 log.cc:824] T 0d22b74aaf404b70b33293e12942fc90 P 5db0920a85d64ed1a68aafcc90dfd515: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:16.111053 731 log.cc:824] T 0d22b74aaf404b70b33293e12942fc90 P 114410c24864480dabd05dc81d0fda31: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:16.111583 733 tablet_bootstrap.cc:654] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:16.112864 733 log.cc:824] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:16.113345 732 tablet_bootstrap.cc:492] T 0d22b74aaf404b70b33293e12942fc90 P 5db0920a85d64ed1a68aafcc90dfd515: No bootstrap required, opened a new log
I20260927 10:57:16.113482 731 tablet_bootstrap.cc:492] T 0d22b74aaf404b70b33293e12942fc90 P 114410c24864480dabd05dc81d0fda31: No bootstrap required, opened a new log
I20260927 10:57:16.113540 732 ts_tablet_manager.cc:1397] T 0d22b74aaf404b70b33293e12942fc90 P 5db0920a85d64ed1a68aafcc90dfd515: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:57:16.113691 731 ts_tablet_manager.cc:1397] T 0d22b74aaf404b70b33293e12942fc90 P 114410c24864480dabd05dc81d0fda31: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:57:16.114677 733 tablet_bootstrap.cc:492] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a: No bootstrap required, opened a new log
I20260927 10:57:16.114838 733 ts_tablet_manager.cc:1397] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:57:16.116863 732 raft_consensus.cc:348] T 0d22b74aaf404b70b33293e12942fc90 P 5db0920a85d64ed1a68aafcc90dfd515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114410c24864480dabd05dc81d0fda31" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41161 } } peers { permanent_uuid: "5db0920a85d64ed1a68aafcc90dfd515" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 39125 } } peers { permanent_uuid: "698d3978bfd8449e8384c65e4b94633a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42331 } }
I20260927 10:57:16.117092 732 raft_consensus.cc:374] T 0d22b74aaf404b70b33293e12942fc90 P 5db0920a85d64ed1a68aafcc90dfd515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:16.117146 732 raft_consensus.cc:729] T 0d22b74aaf404b70b33293e12942fc90 P 5db0920a85d64ed1a68aafcc90dfd515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5db0920a85d64ed1a68aafcc90dfd515, State: Initialized, Role: FOLLOWER
I20260927 10:57:16.117466 731 raft_consensus.cc:348] T 0d22b74aaf404b70b33293e12942fc90 P 114410c24864480dabd05dc81d0fda31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114410c24864480dabd05dc81d0fda31" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41161 } } peers { permanent_uuid: "5db0920a85d64ed1a68aafcc90dfd515" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 39125 } } peers { permanent_uuid: "698d3978bfd8449e8384c65e4b94633a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42331 } }
I20260927 10:57:16.117626 732 consensus_queue.cc:260] T 0d22b74aaf404b70b33293e12942fc90 P 5db0920a85d64ed1a68aafcc90dfd515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114410c24864480dabd05dc81d0fda31" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41161 } } peers { permanent_uuid: "5db0920a85d64ed1a68aafcc90dfd515" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 39125 } } peers { permanent_uuid: "698d3978bfd8449e8384c65e4b94633a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42331 } }
I20260927 10:57:16.117760 731 raft_consensus.cc:374] T 0d22b74aaf404b70b33293e12942fc90 P 114410c24864480dabd05dc81d0fda31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:16.117808 731 raft_consensus.cc:729] T 0d22b74aaf404b70b33293e12942fc90 P 114410c24864480dabd05dc81d0fda31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 114410c24864480dabd05dc81d0fda31, State: Initialized, Role: FOLLOWER
I20260927 10:57:16.118187 731 consensus_queue.cc:260] T 0d22b74aaf404b70b33293e12942fc90 P 114410c24864480dabd05dc81d0fda31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114410c24864480dabd05dc81d0fda31" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41161 } } peers { permanent_uuid: "5db0920a85d64ed1a68aafcc90dfd515" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 39125 } } peers { permanent_uuid: "698d3978bfd8449e8384c65e4b94633a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42331 } }
I20260927 10:57:16.118422 580 heartbeater.cc:503] Master 127.14.68.62:44971 was elected leader, sending a full tablet report...
I20260927 10:57:16.118618 732 ts_tablet_manager.cc:1428] T 0d22b74aaf404b70b33293e12942fc90 P 5db0920a85d64ed1a68aafcc90dfd515: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:57:16.118619 733 raft_consensus.cc:348] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114410c24864480dabd05dc81d0fda31" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41161 } } peers { permanent_uuid: "5db0920a85d64ed1a68aafcc90dfd515" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 39125 } } peers { permanent_uuid: "698d3978bfd8449e8384c65e4b94633a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42331 } }
I20260927 10:57:16.118847 733 raft_consensus.cc:374] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:16.118891 733 raft_consensus.cc:729] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698d3978bfd8449e8384c65e4b94633a, State: Initialized, Role: FOLLOWER
I20260927 10:57:16.119122 731 ts_tablet_manager.cc:1428] T 0d22b74aaf404b70b33293e12942fc90 P 114410c24864480dabd05dc81d0fda31: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:57:16.119268 733 consensus_queue.cc:260] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114410c24864480dabd05dc81d0fda31" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41161 } } peers { permanent_uuid: "5db0920a85d64ed1a68aafcc90dfd515" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 39125 } } peers { permanent_uuid: "698d3978bfd8449e8384c65e4b94633a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42331 } }
I20260927 10:57:16.119416 446 heartbeater.cc:503] Master 127.14.68.62:44971 was elected leader, sending a full tablet report...
I20260927 10:57:16.120155 733 ts_tablet_manager.cc:1428] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:57:16.120447 714 heartbeater.cc:503] Master 127.14.68.62:44971 was elected leader, sending a full tablet report...
W20260927 10:57:16.125022 447 tablet.cc:2369] T 0d22b74aaf404b70b33293e12942fc90 P 114410c24864480dabd05dc81d0fda31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:57:16.160627 739 raft_consensus.cc:482] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:57:16.160815 739 raft_consensus.cc:504] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114410c24864480dabd05dc81d0fda31" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41161 } } peers { permanent_uuid: "5db0920a85d64ed1a68aafcc90dfd515" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 39125 } } peers { permanent_uuid: "698d3978bfd8449e8384c65e4b94633a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42331 } }
I20260927 10:57:16.161819 739 leader_election.cc:288] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 114410c24864480dabd05dc81d0fda31 (127.14.68.1:41161), 5db0920a85d64ed1a68aafcc90dfd515 (127.14.68.2:39125)
I20260927 10:57:16.164395 738 raft_consensus.cc:482] T 0d22b74aaf404b70b33293e12942fc90 P 114410c24864480dabd05dc81d0fda31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:57:16.164561 738 raft_consensus.cc:504] T 0d22b74aaf404b70b33293e12942fc90 P 114410c24864480dabd05dc81d0fda31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114410c24864480dabd05dc81d0fda31" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41161 } } peers { permanent_uuid: "5db0920a85d64ed1a68aafcc90dfd515" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 39125 } } peers { permanent_uuid: "698d3978bfd8449e8384c65e4b94633a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42331 } }
I20260927 10:57:16.166177 738 leader_election.cc:288] T 0d22b74aaf404b70b33293e12942fc90 P 114410c24864480dabd05dc81d0fda31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5db0920a85d64ed1a68aafcc90dfd515 (127.14.68.2:39125), 698d3978bfd8449e8384c65e4b94633a (127.14.68.3:42331)
I20260927 10:57:16.166872 716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.162039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:41161 (local address 127.14.68.3:47281)
0927 10:57:16.162139 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:16.162156 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:16.162243 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:16.162655 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:16.162670 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:16.162688 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:16.162812 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.162817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.163912 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.163915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.165457 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.165464 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.166430 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.166435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.166546 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.166569 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:57:16.166641 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 10:57:16.166728 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":159}
I20260927 10:57:16.167339 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.162277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:47281 (local address 127.14.68.1:41161)
0927 10:57:16.162423 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:16.162426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:16.162438 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:57:16.162484 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:16.162486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:16.162521 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:16.162624 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:16.162951 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.163813 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.165596 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.166229 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.166326 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.166373 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:16.167128 (+ 755us) server_negotiation.cc:299] Negotiation successful
0927 10:57:16.167202 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:57:16.167537 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.162151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:40369 (local address 127.14.68.2:39125)
0927 10:57:16.162289 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:16.162294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:16.162592 (+ 298us) server_negotiation.cc:407] Connection header received
0927 10:57:16.162671 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:16.162674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:16.162723 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:16.162791 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:16.163088 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.163875 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.165596 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.166229 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.166312 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.166370 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:16.167320 (+ 950us) server_negotiation.cc:299] Negotiation successful
0927 10:57:16.167420 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":50}
I20260927 10:57:16.167817 400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d22b74aaf404b70b33293e12942fc90" candidate_uuid: "698d3978bfd8449e8384c65e4b94633a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "114410c24864480dabd05dc81d0fda31" is_pre_election: true
I20260927 10:57:16.168016 400 raft_consensus.cc:2454] T 0d22b74aaf404b70b33293e12942fc90 P 114410c24864480dabd05dc81d0fda31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 698d3978bfd8449e8384c65e4b94633a in term 0.
I20260927 10:57:16.168285 740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.162039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:39125 (local address 127.14.68.3:40369)
0927 10:57:16.162492 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:16.162502 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:16.162576 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:16.162816 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:16.162818 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:16.162850 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:16.162983 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.162987 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.163970 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.163972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.165453 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.165464 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.167068 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.167075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.167183 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.167204 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:57:16.167370 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 10:57:16.167443 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":115,"thread_start_us":147,"threads_started":1}
I20260927 10:57:16.168485 601 leader_election.cc:302] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 114410c24864480dabd05dc81d0fda31, 698d3978bfd8449e8384c65e4b94633a; no voters:
I20260927 10:57:16.168772 739 raft_consensus.cc:2781] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:57:16.168866 739 raft_consensus.cc:482] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:57:16.168913 739 raft_consensus.cc:3037] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:16.169016 534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d22b74aaf404b70b33293e12942fc90" candidate_uuid: "698d3978bfd8449e8384c65e4b94633a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db0920a85d64ed1a68aafcc90dfd515" is_pre_election: true
I20260927 10:57:16.169298 534 raft_consensus.cc:2454] T 0d22b74aaf404b70b33293e12942fc90 P 5db0920a85d64ed1a68aafcc90dfd515 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 698d3978bfd8449e8384c65e4b94633a in term 0.
I20260927 10:57:16.170217 739 raft_consensus.cc:504] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114410c24864480dabd05dc81d0fda31" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41161 } } peers { permanent_uuid: "5db0920a85d64ed1a68aafcc90dfd515" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 39125 } } peers { permanent_uuid: "698d3978bfd8449e8384c65e4b94633a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42331 } }
I20260927 10:57:16.170557 739 leader_election.cc:288] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a [CANDIDATE]: Term 1 election: Requested vote from peers 114410c24864480dabd05dc81d0fda31 (127.14.68.1:41161), 5db0920a85d64ed1a68aafcc90dfd515 (127.14.68.2:39125)
I20260927 10:57:16.171084 400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d22b74aaf404b70b33293e12942fc90" candidate_uuid: "698d3978bfd8449e8384c65e4b94633a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "114410c24864480dabd05dc81d0fda31"
I20260927 10:57:16.171123 742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.165835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:32789 (local address 127.14.68.2:39125)
0927 10:57:16.166747 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:16.166756 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:16.166768 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:57:16.166805 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:16.166810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:16.166852 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:16.167223 (+ 371us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:16.167319 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.168447 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.169934 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.170567 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.170690 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.170739 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:16.170930 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 10:57:16.170994 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":831,"thread_start_us":138,"threads_started":1}
I20260927 10:57:16.171133 741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.165678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:39125 (local address 127.14.68.1:32789)
0927 10:57:16.166405 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:16.166420 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:16.166487 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:16.166966 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:16.166969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:16.166997 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:16.167115 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.167124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.168576 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.168579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.169778 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.169789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.170701 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.170705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.170808 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.170829 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:57:16.170902 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 10:57:16.170988 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":283,"thread_start_us":104,"threads_started":1}
I20260927 10:57:16.171190 400 raft_consensus.cc:3037] T 0d22b74aaf404b70b33293e12942fc90 P 114410c24864480dabd05dc81d0fda31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:16.171689 533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d22b74aaf404b70b33293e12942fc90" candidate_uuid: "114410c24864480dabd05dc81d0fda31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db0920a85d64ed1a68aafcc90dfd515" is_pre_election: true
I20260927 10:57:16.171830 533 raft_consensus.cc:2454] T 0d22b74aaf404b70b33293e12942fc90 P 5db0920a85d64ed1a68aafcc90dfd515 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 114410c24864480dabd05dc81d0fda31 in term 0.
I20260927 10:57:16.171950 730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.166471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:56397 (local address 127.14.68.3:42331)
0927 10:57:16.166587 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:16.166592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:16.166605 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:57:16.166643 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:16.166645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:16.166682 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:16.166748 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:16.167287 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.168004 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.170581 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.171140 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.171230 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.171286 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:16.171793 (+ 507us) server_negotiation.cc:299] Negotiation successful
0927 10:57:16.171854 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 10:57:16.171975 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.165680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:42331 (local address 127.14.68.1:56397)
0927 10:57:16.166225 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:16.166240 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:16.166390 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:16.166860 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:16.166864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:16.166888 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:16.167172 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.167179 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.168501 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.168506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.170447 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.170456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.171512 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.171516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.171620 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.171651 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 10:57:16.171774 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 10:57:16.171842 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":81,"threads_started":1}
I20260927 10:57:16.172192 336 leader_election.cc:302] T 0d22b74aaf404b70b33293e12942fc90 P 114410c24864480dabd05dc81d0fda31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 114410c24864480dabd05dc81d0fda31, 5db0920a85d64ed1a68aafcc90dfd515; no voters:
I20260927 10:57:16.172437 400 raft_consensus.cc:2454] T 0d22b74aaf404b70b33293e12942fc90 P 114410c24864480dabd05dc81d0fda31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 698d3978bfd8449e8384c65e4b94633a in term 1.
I20260927 10:57:16.172438 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d22b74aaf404b70b33293e12942fc90" candidate_uuid: "114410c24864480dabd05dc81d0fda31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "698d3978bfd8449e8384c65e4b94633a" is_pre_election: true
I20260927 10:57:16.172497 738 raft_consensus.cc:2741] T 0d22b74aaf404b70b33293e12942fc90 P 114410c24864480dabd05dc81d0fda31 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260927 10:57:16.172634 668 raft_consensus.cc:2379] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 114410c24864480dabd05dc81d0fda31 in current term 1: Already voted for candidate 698d3978bfd8449e8384c65e4b94633a in this term.
I20260927 10:57:16.172720 534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d22b74aaf404b70b33293e12942fc90" candidate_uuid: "698d3978bfd8449e8384c65e4b94633a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db0920a85d64ed1a68aafcc90dfd515"
I20260927 10:57:16.172762 601 leader_election.cc:302] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 114410c24864480dabd05dc81d0fda31, 698d3978bfd8449e8384c65e4b94633a; no voters:
I20260927 10:57:16.172812 534 raft_consensus.cc:3037] T 0d22b74aaf404b70b33293e12942fc90 P 5db0920a85d64ed1a68aafcc90dfd515 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:16.172901 739 raft_consensus.cc:2781] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:16.173144 739 raft_consensus.cc:686] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a [term 1 LEADER]: Becoming Leader. State: Replica: 698d3978bfd8449e8384c65e4b94633a, State: Running, Role: LEADER
I20260927 10:57:16.173451 739 consensus_queue.cc:237] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114410c24864480dabd05dc81d0fda31" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41161 } } peers { permanent_uuid: "5db0920a85d64ed1a68aafcc90dfd515" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 39125 } } peers { permanent_uuid: "698d3978bfd8449e8384c65e4b94633a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42331 } }
I20260927 10:57:16.174036 534 raft_consensus.cc:2454] T 0d22b74aaf404b70b33293e12942fc90 P 5db0920a85d64ed1a68aafcc90dfd515 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 698d3978bfd8449e8384c65e4b94633a in term 1.
I20260927 10:57:16.176424 32722 catalog_manager.cc:5495] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a reported cstate change: term changed from 0 to 1, leader changed from <none> to 698d3978bfd8449e8384c65e4b94633a (127.14.68.3). New cstate: current_term: 1 leader_uuid: "698d3978bfd8449e8384c65e4b94633a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114410c24864480dabd05dc81d0fda31" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5db0920a85d64ed1a68aafcc90dfd515" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 39125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "698d3978bfd8449e8384c65e4b94633a" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42331 } health_report { overall_health: HEALTHY } } }
I20260927 10:57:16.194818 730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.191286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41130 (local address 127.14.68.3:42331)
0927 10:57:16.191398 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:16.191402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:16.191463 (+ 61us) server_negotiation.cc:407] Connection header received
0927 10:57:16.191581 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:16.191583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:16.191629 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:16.191787 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:57:16.192053 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.192526 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.193449 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.193660 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.193765 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.194061 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:57:16.194423 (+ 362us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:57:16.194573 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:16.194663 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 10:57:16.194694 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 10:57:16.200016 400 raft_consensus.cc:1261] T 0d22b74aaf404b70b33293e12942fc90 P 114410c24864480dabd05dc81d0fda31 [term 1 FOLLOWER]: Refusing update from remote peer 698d3978bfd8449e8384c65e4b94633a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:16.200017 534 raft_consensus.cc:1261] T 0d22b74aaf404b70b33293e12942fc90 P 5db0920a85d64ed1a68aafcc90dfd515 [term 1 FOLLOWER]: Refusing update from remote peer 698d3978bfd8449e8384c65e4b94633a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:16.200695 739 consensus_queue.cc:1035] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db0920a85d64ed1a68aafcc90dfd515" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:16.200913 744 consensus_queue.cc:1035] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a [LEADER]: Connected to new peer: Peer: permanent_uuid: "114410c24864480dabd05dc81d0fda31" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:16.209234 748 mvcc.cc:205] Tried to move back new op lower bound from 7333915181868990464 to 7333915181768646656. Current Snapshot: MvccSnapshot[applied={T|T < 7333915181868990464}]
I20260927 10:57:16.209780 749 mvcc.cc:205] Tried to move back new op lower bound from 7333915181868990464 to 7333915181768646656. Current Snapshot: MvccSnapshot[applied={T|T < 7333915181868990464}]
I20260927 10:57:16.215782 32722 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:57:16.215929 32722 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:57:16.218246 32722 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 610bcbbea88542f5bfac8e0bc175d005: Sending DeleteTablet for 3 replicas of tablet 0d22b74aaf404b70b33293e12942fc90
I20260927 10:57:16.218916 514 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d22b74aaf404b70b33293e12942fc90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:16 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:43640
I20260927 10:57:16.218990 648 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d22b74aaf404b70b33293e12942fc90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:16 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:41122
I20260927 10:57:16.219221 380 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d22b74aaf404b70b33293e12942fc90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:16 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:52802
I20260927 10:57:16.219664 759 tablet_replica.cc:330] T 0d22b74aaf404b70b33293e12942fc90 P 114410c24864480dabd05dc81d0fda31: stopping tablet replica
I20260927 10:57:16.219863 757 tablet_replica.cc:330] T 0d22b74aaf404b70b33293e12942fc90 P 5db0920a85d64ed1a68aafcc90dfd515: stopping tablet replica
I20260927 10:57:16.219975 759 raft_consensus.cc:2229] T 0d22b74aaf404b70b33293e12942fc90 P 114410c24864480dabd05dc81d0fda31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:57:16.220074 757 raft_consensus.cc:2229] T 0d22b74aaf404b70b33293e12942fc90 P 5db0920a85d64ed1a68aafcc90dfd515 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:57:16.220257 759 raft_consensus.cc:2258] T 0d22b74aaf404b70b33293e12942fc90 P 114410c24864480dabd05dc81d0fda31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:16.220273 757 raft_consensus.cc:2258] T 0d22b74aaf404b70b33293e12942fc90 P 5db0920a85d64ed1a68aafcc90dfd515 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:16.220451 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 317
I20260927 10:57:16.222050 757 ts_tablet_manager.cc:1905] T 0d22b74aaf404b70b33293e12942fc90 P 5db0920a85d64ed1a68aafcc90dfd515: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:57:16.224711 757 ts_tablet_manager.cc:1918] T 0d22b74aaf404b70b33293e12942fc90 P 5db0920a85d64ed1a68aafcc90dfd515: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:57:16.224804 757 log.cc:1196] T 0d22b74aaf404b70b33293e12942fc90 P 5db0920a85d64ed1a68aafcc90dfd515: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/0d22b74aaf404b70b33293e12942fc90
I20260927 10:57:16.225113 757 ts_tablet_manager.cc:1939] T 0d22b74aaf404b70b33293e12942fc90 P 5db0920a85d64ed1a68aafcc90dfd515: Deleting consensus metadata
I20260927 10:57:16.226356 32710 catalog_manager.cc:4841] TS 5db0920a85d64ed1a68aafcc90dfd515 (127.14.68.2:39125): tablet 0d22b74aaf404b70b33293e12942fc90 (table test-table [id=1d504072a318402f9957a898bc2e9438]) successfully deleted
I20260927 10:57:16.227959 758 tablet_replica.cc:330] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a: stopping tablet replica
I20260927 10:57:16.228225 758 raft_consensus.cc:2229] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:57:16.228570 758 raft_consensus.cc:2258] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 10:57:16.229818 32707 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.68.1:41161 (error 108)
I20260927 10:57:16.229902 758 ts_tablet_manager.cc:1905] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:57:16.230561 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 451
I20260927 10:57:16.230778 728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.230434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:41161 (local address 127.0.0.1:52806)
0927 10:57:16.230606 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:16.230686 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.1:41161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260927 10:57:16.231014 32707 catalog_manager.cc:4584] TS 114410c24864480dabd05dc81d0fda31 (127.14.68.1:41161): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0d22b74aaf404b70b33293e12942fc90: Network error: Client connection negotiation failed: client connection to 127.14.68.1:41161: connect: Connection refused (error 111)
I20260927 10:57:16.232376 758 ts_tablet_manager.cc:1918] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:57:16.232462 758 log.cc:1196] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/0d22b74aaf404b70b33293e12942fc90
I20260927 10:57:16.232774 758 ts_tablet_manager.cc:1939] T 0d22b74aaf404b70b33293e12942fc90 P 698d3978bfd8449e8384c65e4b94633a: Deleting consensus metadata
I20260927 10:57:16.233776 32707 catalog_manager.cc:4841] TS 698d3978bfd8449e8384c65e4b94633a (127.14.68.3:42331): tablet 0d22b74aaf404b70b33293e12942fc90 (table test-table [id=1d504072a318402f9957a898bc2e9438]) successfully deleted
I20260927 10:57:16.237504 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 585
I20260927 10:57:16.243266 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 32691
2026-09-27T10:57:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2210 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T10:57:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:16Z Disabled control of system clock
I20260927 10:57:16.272439 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:42007
--webserver_interface=127.14.68.62
--webserver_port=0
--builtin_ntp_servers=127.14.68.20:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:42007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:57:16.379143 764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:16.379406 764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:16.379441 764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:16.383388 764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:16.383443 764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:16.383457 764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:16.383490 764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:16.383505 764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:16.388315 764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:42007
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:42007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:16.389319 764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:16.390627 764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:57:16.398933 764 server_base.cc:1034] running on GCE node
W20260927 10:57:16.398964 770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:16.398913 769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:16.398943 772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:16.399626 764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:16.400669 764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:16.401882 764 hybrid_clock.cc:648] HybridClock initialized: now 1790506636401868 us; error 31 us; skew 500 ppm
I20260927 10:57:16.403656 764 webserver.cc:460] Webserver started at http://127.14.68.62:42583/ using document root <none> and password file <none>
I20260927 10:57:16.404258 764 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:16.404316 764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:16.404557 764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:16.406477 764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "17f2deeed1cb4d5794ff46ac4b1fe1bd"
format_stamp: "Formatted at 2026-09-27 10:57:16 on dist-test-slave-tz51"
server_key: "ee4e392fcb4eee111126178309b73083"
server_key_iv: "24736c6b603075622a5c2960564e253e"
server_key_version: "encryptionkey@0"
I20260927 10:57:16.407047 764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "17f2deeed1cb4d5794ff46ac4b1fe1bd"
format_stamp: "Formatted at 2026-09-27 10:57:16 on dist-test-slave-tz51"
server_key: "ee4e392fcb4eee111126178309b73083"
server_key_iv: "24736c6b603075622a5c2960564e253e"
server_key_version: "encryptionkey@0"
I20260927 10:57:16.410480 764 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:57:16.412734 778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:16.413822 764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:57:16.413980 764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "17f2deeed1cb4d5794ff46ac4b1fe1bd"
format_stamp: "Formatted at 2026-09-27 10:57:16 on dist-test-slave-tz51"
server_key: "ee4e392fcb4eee111126178309b73083"
server_key_iv: "24736c6b603075622a5c2960564e253e"
server_key_version: "encryptionkey@0"
I20260927 10:57:16.414086 764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:16.428898 764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:16.429509 764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:16.429699 764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:16.436380 764 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:42007
I20260927 10:57:16.436383 830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:42007 every 8 connection(s)
I20260927 10:57:16.437249 764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:16.437525 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 764
I20260927 10:57:16.437649 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:16.438017 14608 external_mini_cluster.cc:1428] Setting key c4641305e164c43b3b0c3da9239d1aa9
I20260927 10:57:16.440471 831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:16.446475 831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd: Bootstrap starting.
I20260927 10:57:16.446771 834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.439319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58784 (local address 127.14.68.62:42007)
0927 10:57:16.439924 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:16.439939 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:16.439972 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:57:16.440609 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:16.440630 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:16.441001 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:16.441274 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:16.441578 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.442486 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.443684 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.444197 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.444316 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.445112 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:16.445117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:16.445128 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:16.445481 (+ 353us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:16.445530 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:16.445549 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:16.445622 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:16.445795 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 10:57:16.446078 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":135,"threads_started":1}
I20260927 10:57:16.448555 831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:16.449275 831 log.cc:824] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:16.451092 831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd: No bootstrap required, opened a new log
I20260927 10:57:16.454303 831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f2deeed1cb4d5794ff46ac4b1fe1bd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 42007 } }
I20260927 10:57:16.454519 831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:16.454604 831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17f2deeed1cb4d5794ff46ac4b1fe1bd, State: Initialized, Role: FOLLOWER
I20260927 10:57:16.455040 831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f2deeed1cb4d5794ff46ac4b1fe1bd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 42007 } }
I20260927 10:57:16.455185 831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:16.455253 831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:16.455364 831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:16.456192 831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f2deeed1cb4d5794ff46ac4b1fe1bd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 42007 } }
I20260927 10:57:16.456492 831 leader_election.cc:302] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17f2deeed1cb4d5794ff46ac4b1fe1bd; no voters:
I20260927 10:57:16.456768 831 leader_election.cc:288] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:16.456864 836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:16.457079 836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd [term 1 LEADER]: Becoming Leader. State: Replica: 17f2deeed1cb4d5794ff46ac4b1fe1bd, State: Running, Role: LEADER
I20260927 10:57:16.457381 836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f2deeed1cb4d5794ff46ac4b1fe1bd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 42007 } }
I20260927 10:57:16.457911 831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:16.458969 838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17f2deeed1cb4d5794ff46ac4b1fe1bd. Latest consensus state: current_term: 1 leader_uuid: "17f2deeed1cb4d5794ff46ac4b1fe1bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f2deeed1cb4d5794ff46ac4b1fe1bd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 42007 } } }
I20260927 10:57:16.459089 838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:16.459350 837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17f2deeed1cb4d5794ff46ac4b1fe1bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f2deeed1cb4d5794ff46ac4b1fe1bd" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 42007 } } }
I20260927 10:57:16.459444 837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:16.459570 845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:16.462210 845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:16.467464 845 catalog_manager.cc:1347] Generated new cluster ID: f03a37cf1c7146d7b0d6feef26d6fede
I20260927 10:57:16.467528 845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:16.479465 845 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:16.480391 845 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:16.487064 845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd: Generated new TSK 0
I20260927 10:57:16.487742 845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:57:16.495889 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:42007
--builtin_ntp_servers=127.14.68.20:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:57:16.603730 855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:16.604035 855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:16.604075 855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:16.607931 855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:16.607975 855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:16.608026 855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:57:16.612607 855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:42007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:16.613526 855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:16.614784 855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:16.623936 860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:16.624028 855 server_base.cc:1034] running on GCE node
W20260927 10:57:16.623979 861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:16.623936 863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:16.624558 855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:16.625151 855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:16.626327 855 hybrid_clock.cc:648] HybridClock initialized: now 1790506636626281 us; error 38 us; skew 500 ppm
I20260927 10:57:16.628345 855 webserver.cc:460] Webserver started at http://127.14.68.1:36211/ using document root <none> and password file <none>
I20260927 10:57:16.628837 855 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:16.628894 855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:16.629043 855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:16.630918 855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "e3eccc80d4134e2aaad313b4682f140f"
format_stamp: "Formatted at 2026-09-27 10:57:16 on dist-test-slave-tz51"
server_key: "a90bd041b23328a1bd4828ce671dcfcf"
server_key_iv: "0d45e7ce4f49c567b864c7caae316dc1"
server_key_version: "encryptionkey@0"
I20260927 10:57:16.631421 855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "e3eccc80d4134e2aaad313b4682f140f"
format_stamp: "Formatted at 2026-09-27 10:57:16 on dist-test-slave-tz51"
server_key: "a90bd041b23328a1bd4828ce671dcfcf"
server_key_iv: "0d45e7ce4f49c567b864c7caae316dc1"
server_key_version: "encryptionkey@0"
I20260927 10:57:16.634895 855 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 10:57:16.637311 869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:16.638440 855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 10:57:16.638564 855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "e3eccc80d4134e2aaad313b4682f140f"
format_stamp: "Formatted at 2026-09-27 10:57:16 on dist-test-slave-tz51"
server_key: "a90bd041b23328a1bd4828ce671dcfcf"
server_key_iv: "0d45e7ce4f49c567b864c7caae316dc1"
server_key_version: "encryptionkey@0"
I20260927 10:57:16.638689 855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:16.649497 855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:16.650128 855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:16.650322 855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:16.650887 855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:16.651829 855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:16.651875 855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:16.651938 855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:16.651969 855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:16.661494 855 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:33651
I20260927 10:57:16.661518 982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:33651 every 8 connection(s)
I20260927 10:57:16.662356 855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:57:16.671566 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 855
I20260927 10:57:16.671747 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:57:16.672039 14608 external_mini_cluster.cc:1428] Setting key 8321fa6b9819028b976202e44d37e5e5
I20260927 10:57:16.672784 834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.664183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:50685 (local address 127.14.68.62:42007)
0927 10:57:16.664346 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:16.664351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:16.665688 (+ 1337us) server_negotiation.cc:407] Connection header received
0927 10:57:16.667457 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:16.667461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:16.667526 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:16.667639 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:16.668750 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.669442 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.670657 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.670887 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.670999 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.672242 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:16.672246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:16.672248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:16.672268 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:16.672288 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:16.672295 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:16.672373 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:16.672541 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 10:57:16.672640 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 10:57:16.673659 985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.664631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:42007 (local address 127.14.68.1:50685)
0927 10:57:16.665512 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:16.665576 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:16.667228 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:16.667861 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:16.667868 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:16.667974 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:16.668605 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.668619 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.669615 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.669618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.670512 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.670521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.671038 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.671041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.671213 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.671938 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:57:16.671939 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:16.671979 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:57:16.672041 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:16.672386 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:16.672396 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:16.672402 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:57:16.672529 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 10:57:16.672796 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":155,"threads_started":1}
I20260927 10:57:16.674041 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:42007
--builtin_ntp_servers=127.14.68.20:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:57:16.674860 983 heartbeater.cc:347] Connected to a master server at 127.14.68.62:42007
I20260927 10:57:16.675112 983 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:16.675611 983 heartbeater.cc:511] Master 127.14.68.62:42007 requested a full tablet report, sending...
I20260927 10:57:16.677165 795 ts_manager.cc:194] Registered new tserver with Master: e3eccc80d4134e2aaad313b4682f140f (127.14.68.1:33651)
I20260927 10:57:16.678855 795 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.68.1:50685
W20260927 10:57:16.780941 986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:16.781193 986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:16.781229 986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:16.785058 986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:16.785115 986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:16.785166 986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:57:16.789685 986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:42007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:16.790642 986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:16.791925 986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:16.801149 991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:16.801182 986 server_base.cc:1034] running on GCE node
W20260927 10:57:16.801152 994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:16.801147 992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:16.801723 986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:16.802349 986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:16.803503 986 hybrid_clock.cc:648] HybridClock initialized: now 1790506636803478 us; error 39 us; skew 500 ppm
I20260927 10:57:16.805200 986 webserver.cc:460] Webserver started at http://127.14.68.2:39307/ using document root <none> and password file <none>
I20260927 10:57:16.805738 986 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:16.805797 986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:16.806054 986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:16.808058 986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "86ef347370b84c4db0dd513e68ae51be"
format_stamp: "Formatted at 2026-09-27 10:57:16 on dist-test-slave-tz51"
server_key: "74b65d77f818881a46c5957c081814ec"
server_key_iv: "e9e348c63cc91d241b7b85667c405912"
server_key_version: "encryptionkey@0"
I20260927 10:57:16.808599 986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "86ef347370b84c4db0dd513e68ae51be"
format_stamp: "Formatted at 2026-09-27 10:57:16 on dist-test-slave-tz51"
server_key: "74b65d77f818881a46c5957c081814ec"
server_key_iv: "e9e348c63cc91d241b7b85667c405912"
server_key_version: "encryptionkey@0"
I20260927 10:57:16.813283 986 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 10:57:16.817049 1000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:16.818418 986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:57:16.818511 986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "86ef347370b84c4db0dd513e68ae51be"
format_stamp: "Formatted at 2026-09-27 10:57:16 on dist-test-slave-tz51"
server_key: "74b65d77f818881a46c5957c081814ec"
server_key_iv: "e9e348c63cc91d241b7b85667c405912"
server_key_version: "encryptionkey@0"
I20260927 10:57:16.818595 986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:16.831593 986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:16.832322 986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:16.832474 986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:16.833000 986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:16.833880 986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:16.833925 986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:16.833964 986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:16.833985 986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:16.842617 986 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:46311
I20260927 10:57:16.842664 1113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:46311 every 8 connection(s)
I20260927 10:57:16.843582 986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:57:16.850193 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 986
I20260927 10:57:16.850347 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:57:16.850631 14608 external_mini_cluster.cc:1428] Setting key 5e9c775dd232a2306cefbf5622323ec6
I20260927 10:57:16.852495 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:42007
--builtin_ntp_servers=127.14.68.20:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:57:16.853205 834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.845477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:32917 (local address 127.14.68.62:42007)
0927 10:57:16.845630 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:16.845634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:16.846696 (+ 1062us) server_negotiation.cc:407] Connection header received
0927 10:57:16.848152 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:16.848155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:16.848217 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:16.848296 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:16.849193 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.849721 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.851010 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.851315 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.851414 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.852585 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:16.852589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:16.852591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:16.852605 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:16.852627 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:16.852637 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:16.852706 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:16.852906 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 10:57:16.852977 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 10:57:16.854001 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.845878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:42007 (local address 127.14.68.2:32917)
0927 10:57:16.846536 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:16.846575 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:16.847921 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:16.848475 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:16.848482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:16.848566 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:16.849041 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.849053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.849840 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.849843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.850890 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.850900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.851449 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.851453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.851618 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.852333 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:57:16.852334 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:16.852373 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:57:16.852432 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:16.852743 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:16.852751 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:16.852756 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 10:57:16.852864 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 10:57:16.853078 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":138,"threads_started":1}
I20260927 10:57:16.855175 1114 heartbeater.cc:347] Connected to a master server at 127.14.68.62:42007
I20260927 10:57:16.855412 1114 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:16.855965 1114 heartbeater.cc:511] Master 127.14.68.62:42007 requested a full tablet report, sending...
I20260927 10:57:16.856982 795 ts_manager.cc:194] Registered new tserver with Master: 86ef347370b84c4db0dd513e68ae51be (127.14.68.2:46311)
I20260927 10:57:16.857841 795 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.68.2:32917
W20260927 10:57:16.962116 1117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:16.962349 1117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:16.962385 1117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:16.966282 1117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:16.966333 1117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:16.966387 1117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:57:16.971032 1117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:42007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:16.972070 1117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:16.973336 1117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:16.982250 1123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:16.982259 1122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:16.982331 1117 server_base.cc:1034] running on GCE node
W20260927 10:57:16.982259 1125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:16.982858 1117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:16.983451 1117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:16.984606 1117 hybrid_clock.cc:648] HybridClock initialized: now 1790506636984589 us; error 26 us; skew 500 ppm
I20260927 10:57:16.986289 1117 webserver.cc:460] Webserver started at http://127.14.68.3:39765/ using document root <none> and password file <none>
I20260927 10:57:16.986943 1117 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:16.987052 1117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:16.987270 1117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:16.989269 1117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "e47a8f0c57a543438001b6d716492740"
format_stamp: "Formatted at 2026-09-27 10:57:16 on dist-test-slave-tz51"
server_key: "db42bcd87075e40de80cec6073e92266"
server_key_iv: "ee9261d308ab0c4dc5f470144dfa2e98"
server_key_version: "encryptionkey@0"
I20260927 10:57:16.989806 1117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "e47a8f0c57a543438001b6d716492740"
format_stamp: "Formatted at 2026-09-27 10:57:16 on dist-test-slave-tz51"
server_key: "db42bcd87075e40de80cec6073e92266"
server_key_iv: "ee9261d308ab0c4dc5f470144dfa2e98"
server_key_version: "encryptionkey@0"
I20260927 10:57:16.993309 1117 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:57:16.995543 1131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:16.996726 1117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:16.996846 1117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "e47a8f0c57a543438001b6d716492740"
format_stamp: "Formatted at 2026-09-27 10:57:16 on dist-test-slave-tz51"
server_key: "db42bcd87075e40de80cec6073e92266"
server_key_iv: "ee9261d308ab0c4dc5f470144dfa2e98"
server_key_version: "encryptionkey@0"
I20260927 10:57:16.996943 1117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:17.022212 1117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:17.022914 1117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:17.023057 1117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:17.023583 1117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:17.024518 1117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:17.024564 1117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:17.024603 1117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:17.024624 1117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:17.033155 1117 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:42893
I20260927 10:57:17.033208 1244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:42893 every 8 connection(s)
I20260927 10:57:17.034044 1117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:57:17.038964 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 1117
I20260927 10:57:17.039059 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:57:17.039400 14608 external_mini_cluster.cc:1428] Setting key f16896f25a5fce27c226c64a59c3084c
I20260927 10:57:17.043637 834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:17.035808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:49017 (local address 127.14.68.62:42007)
0927 10:57:17.035944 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:17.035947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:17.037047 (+ 1100us) server_negotiation.cc:407] Connection header received
0927 10:57:17.038453 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:17.038456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:17.038512 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:17.038617 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:17.039609 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.040321 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.041683 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.041937 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.042045 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:17.043162 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:17.043165 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:17.043168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:17.043179 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:17.043202 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:17.043216 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:17.043293 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:17.043446 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 10:57:17.043504 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:57:17.044507 1247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:17.036287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:42007 (local address 127.14.68.3:49017)
0927 10:57:17.036901 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:17.036940 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:17.038239 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:17.038804 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:17.038810 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:17.038891 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:17.039460 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:17.039482 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.040435 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.040439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:17.041563 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:17.041571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.042088 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.042092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:17.042258 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:17.042917 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:57:17.042918 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:17.042957 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:57:17.043018 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:17.043294 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:17.043306 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:17.043314 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 10:57:17.043445 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 10:57:17.043739 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":108,"threads_started":1}
I20260927 10:57:17.045548 1245 heartbeater.cc:347] Connected to a master server at 127.14.68.62:42007
I20260927 10:57:17.045775 1245 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:17.046271 1245 heartbeater.cc:511] Master 127.14.68.62:42007 requested a full tablet report, sending...
I20260927 10:57:17.047210 795 ts_manager.cc:194] Registered new tserver with Master: e47a8f0c57a543438001b6d716492740 (127.14.68.3:42893)
I20260927 10:57:17.047824 795 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.68.3:49017
I20260927 10:57:17.053241 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:57:17.058521 834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:17.055129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58790 (local address 127.14.68.62:42007)
0927 10:57:17.055249 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:17.055253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:17.055401 (+ 148us) server_negotiation.cc:407] Connection header received
0927 10:57:17.055486 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:17.055487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:17.055523 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:17.055585 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:17.055850 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.056323 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.057074 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.057272 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.057341 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:17.058083 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:17.058085 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:17.058088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:17.058098 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:17.058115 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:17.058123 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:17.058219 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:17.058395 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 10:57:17.058424 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:57:17.061301 795 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:57:17.064597 795 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:57:17.081821 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:17.076320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:42893 (local address 127.0.0.1:52450)
0927 10:57:17.076659 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:17.076711 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:17.076858 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:17.077602 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:17.077607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:17.077625 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:17.077781 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:17.077789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.079015 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.079019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:17.080217 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:17.080226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.081262 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.081271 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:17.081376 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:17.081415 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 10:57:17.081535 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 10:57:17.081653 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":70,"threads_started":1}
I20260927 10:57:17.081832 1257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:17.076464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52450 (local address 127.14.68.3:42893)
0927 10:57:17.076794 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:17.076805 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:17.076832 (+ 27us) server_negotiation.cc:407] Connection header received
0927 10:57:17.077085 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:17.077091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:17.077305 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:17.077454 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:17.077911 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.078892 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.080347 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.081143 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.081251 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:17.081459 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:17.081560 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 10:57:17.081662 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":131,"threads_started":1}
I20260927 10:57:17.082430 1258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:17.077053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:33651 (local address 127.0.0.1:56052)
0927 10:57:17.077278 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:17.077291 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:17.077376 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:17.078237 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:17.078240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:17.078257 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:17.078356 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:17.078363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.079732 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.079735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:17.080844 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:17.080853 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.082018 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.082024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:17.082116 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:17.082144 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 10:57:17.082242 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 10:57:17.082307 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":3,"thread_start_us":100,"threads_started":1}
I20260927 10:57:17.082635 1259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:17.077225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56052 (local address 127.14.68.1:33651)
0927 10:57:17.077625 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:17.077638 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:17.077667 (+ 29us) server_negotiation.cc:407] Connection header received
0927 10:57:17.077746 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:17.077752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:17.077925 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:17.078078 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:17.078481 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.079593 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.080968 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.081674 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.081769 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:17.081957 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:17.082273 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 10:57:17.082363 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":83,"threads_started":1}
I20260927 10:57:17.084918 1179 tablet_service.cc:1467] Processing CreateTablet for tablet 8d20f1cb73284f16947b6658bcab359b (DEFAULT_TABLE table=test-table [id=571354aa3b074900a6da6e1d947a2ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:17.085278 917 tablet_service.cc:1467] Processing CreateTablet for tablet 8d20f1cb73284f16947b6658bcab359b (DEFAULT_TABLE table=test-table [id=571354aa3b074900a6da6e1d947a2ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:17.086069 1179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d20f1cb73284f16947b6658bcab359b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:17.086225 917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d20f1cb73284f16947b6658bcab359b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:17.088174 1260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:17.077354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:46311 (local address 127.0.0.1:51942)
0927 10:57:17.078406 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:17.078421 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:17.078496 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:17.078904 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:17.078907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:17.078924 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:17.078999 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:17.079005 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.080094 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.080098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:17.081267 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:17.081277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.087777 (+ 6500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.087780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:17.087868 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:17.087891 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:57:17.087964 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 10:57:17.088036 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"mutex_wait_us":30,"thread_start_us":70,"threads_started":1}
I20260927 10:57:17.088653 1261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:17.077869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51942 (local address 127.14.68.2:46311)
0927 10:57:17.078275 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:17.078286 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:17.078583 (+ 297us) server_negotiation.cc:407] Connection header received
0927 10:57:17.078644 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:17.078647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:17.078775 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:17.078920 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:17.079117 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.079963 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.081398 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.083777 (+ 2379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.084076 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:17.084207 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:17.088363 (+ 4156us) server_negotiation.cc:299] Negotiation successful
0927 10:57:17.088476 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":128,"threads_started":1}
I20260927 10:57:17.090847 1048 tablet_service.cc:1467] Processing CreateTablet for tablet 8d20f1cb73284f16947b6658bcab359b (DEFAULT_TABLE table=test-table [id=571354aa3b074900a6da6e1d947a2ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:17.091679 1048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d20f1cb73284f16947b6658bcab359b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:17.093593 1262 tablet_bootstrap.cc:492] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f: Bootstrap starting.
I20260927 10:57:17.094746 1263 tablet_bootstrap.cc:492] T 8d20f1cb73284f16947b6658bcab359b P e47a8f0c57a543438001b6d716492740: Bootstrap starting.
I20260927 10:57:17.096015 1262 tablet_bootstrap.cc:654] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:17.096452 1263 tablet_bootstrap.cc:654] T 8d20f1cb73284f16947b6658bcab359b P e47a8f0c57a543438001b6d716492740: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:17.096874 1262 log.cc:824] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:17.097149 1263 log.cc:824] T 8d20f1cb73284f16947b6658bcab359b P e47a8f0c57a543438001b6d716492740: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:17.098383 1266 tablet_bootstrap.cc:492] T 8d20f1cb73284f16947b6658bcab359b P 86ef347370b84c4db0dd513e68ae51be: Bootstrap starting.
I20260927 10:57:17.098889 1262 tablet_bootstrap.cc:492] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f: No bootstrap required, opened a new log
I20260927 10:57:17.098886 1263 tablet_bootstrap.cc:492] T 8d20f1cb73284f16947b6658bcab359b P e47a8f0c57a543438001b6d716492740: No bootstrap required, opened a new log
I20260927 10:57:17.099072 1262 ts_tablet_manager.cc:1397] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:57:17.099072 1263 ts_tablet_manager.cc:1397] T 8d20f1cb73284f16947b6658bcab359b P e47a8f0c57a543438001b6d716492740: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 10:57:17.100351 1266 tablet_bootstrap.cc:654] T 8d20f1cb73284f16947b6658bcab359b P 86ef347370b84c4db0dd513e68ae51be: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:17.100953 1266 log.cc:824] T 8d20f1cb73284f16947b6658bcab359b P 86ef347370b84c4db0dd513e68ae51be: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:17.102624 1266 tablet_bootstrap.cc:492] T 8d20f1cb73284f16947b6658bcab359b P 86ef347370b84c4db0dd513e68ae51be: No bootstrap required, opened a new log
I20260927 10:57:17.102814 1266 ts_tablet_manager.cc:1397] T 8d20f1cb73284f16947b6658bcab359b P 86ef347370b84c4db0dd513e68ae51be: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:57:17.103284 1263 raft_consensus.cc:348] T 8d20f1cb73284f16947b6658bcab359b P e47a8f0c57a543438001b6d716492740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47a8f0c57a543438001b6d716492740" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42893 } } peers { permanent_uuid: "e3eccc80d4134e2aaad313b4682f140f" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33651 } } peers { permanent_uuid: "86ef347370b84c4db0dd513e68ae51be" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 46311 } }
I20260927 10:57:17.103284 1262 raft_consensus.cc:348] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47a8f0c57a543438001b6d716492740" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42893 } } peers { permanent_uuid: "e3eccc80d4134e2aaad313b4682f140f" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33651 } } peers { permanent_uuid: "86ef347370b84c4db0dd513e68ae51be" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 46311 } }
I20260927 10:57:17.103540 1262 raft_consensus.cc:374] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:17.103540 1263 raft_consensus.cc:374] T 8d20f1cb73284f16947b6658bcab359b P e47a8f0c57a543438001b6d716492740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:17.103598 1262 raft_consensus.cc:729] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3eccc80d4134e2aaad313b4682f140f, State: Initialized, Role: FOLLOWER
I20260927 10:57:17.103605 1263 raft_consensus.cc:729] T 8d20f1cb73284f16947b6658bcab359b P e47a8f0c57a543438001b6d716492740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47a8f0c57a543438001b6d716492740, State: Initialized, Role: FOLLOWER
I20260927 10:57:17.104067 1262 consensus_queue.cc:260] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47a8f0c57a543438001b6d716492740" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42893 } } peers { permanent_uuid: "e3eccc80d4134e2aaad313b4682f140f" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33651 } } peers { permanent_uuid: "86ef347370b84c4db0dd513e68ae51be" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 46311 } }
I20260927 10:57:17.104067 1263 consensus_queue.cc:260] T 8d20f1cb73284f16947b6658bcab359b P e47a8f0c57a543438001b6d716492740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47a8f0c57a543438001b6d716492740" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42893 } } peers { permanent_uuid: "e3eccc80d4134e2aaad313b4682f140f" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33651 } } peers { permanent_uuid: "86ef347370b84c4db0dd513e68ae51be" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 46311 } }
I20260927 10:57:17.105014 1262 ts_tablet_manager.cc:1428] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:57:17.105014 1263 ts_tablet_manager.cc:1428] T 8d20f1cb73284f16947b6658bcab359b P e47a8f0c57a543438001b6d716492740: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:57:17.105374 983 heartbeater.cc:503] Master 127.14.68.62:42007 was elected leader, sending a full tablet report...
I20260927 10:57:17.106387 1245 heartbeater.cc:503] Master 127.14.68.62:42007 was elected leader, sending a full tablet report...
I20260927 10:57:17.107242 1266 raft_consensus.cc:348] T 8d20f1cb73284f16947b6658bcab359b P 86ef347370b84c4db0dd513e68ae51be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47a8f0c57a543438001b6d716492740" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42893 } } peers { permanent_uuid: "e3eccc80d4134e2aaad313b4682f140f" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33651 } } peers { permanent_uuid: "86ef347370b84c4db0dd513e68ae51be" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 46311 } }
I20260927 10:57:17.107497 1266 raft_consensus.cc:374] T 8d20f1cb73284f16947b6658bcab359b P 86ef347370b84c4db0dd513e68ae51be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:17.107558 1266 raft_consensus.cc:729] T 8d20f1cb73284f16947b6658bcab359b P 86ef347370b84c4db0dd513e68ae51be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ef347370b84c4db0dd513e68ae51be, State: Initialized, Role: FOLLOWER
I20260927 10:57:17.107986 1266 consensus_queue.cc:260] T 8d20f1cb73284f16947b6658bcab359b P 86ef347370b84c4db0dd513e68ae51be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47a8f0c57a543438001b6d716492740" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42893 } } peers { permanent_uuid: "e3eccc80d4134e2aaad313b4682f140f" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33651 } } peers { permanent_uuid: "86ef347370b84c4db0dd513e68ae51be" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 46311 } }
I20260927 10:57:17.108577 1114 heartbeater.cc:503] Master 127.14.68.62:42007 was elected leader, sending a full tablet report...
I20260927 10:57:17.109591 1266 ts_tablet_manager.cc:1428] T 8d20f1cb73284f16947b6658bcab359b P 86ef347370b84c4db0dd513e68ae51be: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260927 10:57:17.163491 984 tablet.cc:2369] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:57:17.285107 1246 tablet.cc:2369] T 8d20f1cb73284f16947b6658bcab359b P e47a8f0c57a543438001b6d716492740: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:57:17.344825 1115 tablet.cc:2369] T 8d20f1cb73284f16947b6658bcab359b P 86ef347370b84c4db0dd513e68ae51be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:57:17.493338 1269 raft_consensus.cc:482] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:57:17.493615 1269 raft_consensus.cc:504] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47a8f0c57a543438001b6d716492740" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42893 } } peers { permanent_uuid: "e3eccc80d4134e2aaad313b4682f140f" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33651 } } peers { permanent_uuid: "86ef347370b84c4db0dd513e68ae51be" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 46311 } }
I20260927 10:57:17.494803 1269 leader_election.cc:288] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47a8f0c57a543438001b6d716492740 (127.14.68.3:42893), 86ef347370b84c4db0dd513e68ae51be (127.14.68.2:46311)
I20260927 10:57:17.498700 1257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:17.495196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:38739 (local address 127.14.68.3:42893)
0927 10:57:17.495327 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:17.495332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:17.495455 (+ 123us) server_negotiation.cc:407] Connection header received
0927 10:57:17.495658 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:17.495665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:17.495736 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:17.495791 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:17.496168 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.496685 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.497750 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.498146 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.498269 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:17.498328 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:17.498507 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 10:57:17.498577 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 10:57:17.498721 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:17.495029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:42893 (local address 127.14.68.1:38739)
0927 10:57:17.495339 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:17.495355 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:17.495516 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:17.495873 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:17.495877 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:17.495912 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:17.496022 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:17.496027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.496811 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.496814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:17.497617 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:17.497626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.498273 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.498277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:17.498381 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:17.498401 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:57:17.498490 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 10:57:17.498569 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"spinlock_wait_cycles":4736,"thread_start_us":133,"threads_started":1}
I20260927 10:57:17.498881 1261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:17.495605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:60281 (local address 127.14.68.2:46311)
0927 10:57:17.495771 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:17.495777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:17.495862 (+ 85us) server_negotiation.cc:407] Connection header received
0927 10:57:17.496023 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:17.496026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:17.496078 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:17.496168 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:17.496434 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.496921 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.497934 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.498326 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.498445 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:17.498498 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:17.498709 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 10:57:17.498777 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:57:17.498900 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:17.495471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:46311 (local address 127.14.68.1:60281)
0927 10:57:17.495776 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:17.495788 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:17.495910 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:17.496173 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:17.496177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:17.496192 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:17.496325 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:17.496330 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.497035 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.497040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:17.497797 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:17.497805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.498459 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.498463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:17.498558 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:17.498586 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 10:57:17.498690 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 10:57:17.498757 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"spinlock_wait_cycles":1920,"thread_start_us":81,"threads_started":1}
I20260927 10:57:17.499815 1068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d20f1cb73284f16947b6658bcab359b" candidate_uuid: "e3eccc80d4134e2aaad313b4682f140f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ef347370b84c4db0dd513e68ae51be" is_pre_election: true
I20260927 10:57:17.499814 1199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d20f1cb73284f16947b6658bcab359b" candidate_uuid: "e3eccc80d4134e2aaad313b4682f140f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47a8f0c57a543438001b6d716492740" is_pre_election: true
I20260927 10:57:17.500139 1068 raft_consensus.cc:2454] T 8d20f1cb73284f16947b6658bcab359b P 86ef347370b84c4db0dd513e68ae51be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3eccc80d4134e2aaad313b4682f140f in term 0.
I20260927 10:57:17.500134 1199 raft_consensus.cc:2454] T 8d20f1cb73284f16947b6658bcab359b P e47a8f0c57a543438001b6d716492740 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3eccc80d4134e2aaad313b4682f140f in term 0.
I20260927 10:57:17.500634 872 leader_election.cc:302] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3eccc80d4134e2aaad313b4682f140f, e47a8f0c57a543438001b6d716492740; no voters:
I20260927 10:57:17.500880 1269 raft_consensus.cc:2781] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:57:17.500965 1269 raft_consensus.cc:482] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:57:17.501008 1269 raft_consensus.cc:3037] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:17.502063 1269 raft_consensus.cc:504] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47a8f0c57a543438001b6d716492740" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42893 } } peers { permanent_uuid: "e3eccc80d4134e2aaad313b4682f140f" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33651 } } peers { permanent_uuid: "86ef347370b84c4db0dd513e68ae51be" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 46311 } }
I20260927 10:57:17.502388 1269 leader_election.cc:288] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f [CANDIDATE]: Term 1 election: Requested vote from peers e47a8f0c57a543438001b6d716492740 (127.14.68.3:42893), 86ef347370b84c4db0dd513e68ae51be (127.14.68.2:46311)
I20260927 10:57:17.502703 1199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d20f1cb73284f16947b6658bcab359b" candidate_uuid: "e3eccc80d4134e2aaad313b4682f140f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47a8f0c57a543438001b6d716492740"
I20260927 10:57:17.502733 1068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d20f1cb73284f16947b6658bcab359b" candidate_uuid: "e3eccc80d4134e2aaad313b4682f140f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ef347370b84c4db0dd513e68ae51be"
I20260927 10:57:17.502858 1199 raft_consensus.cc:3037] T 8d20f1cb73284f16947b6658bcab359b P e47a8f0c57a543438001b6d716492740 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:17.502887 1068 raft_consensus.cc:3037] T 8d20f1cb73284f16947b6658bcab359b P 86ef347370b84c4db0dd513e68ae51be [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:17.503925 1199 raft_consensus.cc:2454] T 8d20f1cb73284f16947b6658bcab359b P e47a8f0c57a543438001b6d716492740 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3eccc80d4134e2aaad313b4682f140f in term 1.
I20260927 10:57:17.504016 1068 raft_consensus.cc:2454] T 8d20f1cb73284f16947b6658bcab359b P 86ef347370b84c4db0dd513e68ae51be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3eccc80d4134e2aaad313b4682f140f in term 1.
I20260927 10:57:17.504289 872 leader_election.cc:302] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3eccc80d4134e2aaad313b4682f140f, e47a8f0c57a543438001b6d716492740; no voters:
I20260927 10:57:17.504451 1269 raft_consensus.cc:2781] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:17.504678 1269 raft_consensus.cc:686] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f [term 1 LEADER]: Becoming Leader. State: Replica: e3eccc80d4134e2aaad313b4682f140f, State: Running, Role: LEADER
I20260927 10:57:17.505021 1269 consensus_queue.cc:237] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47a8f0c57a543438001b6d716492740" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42893 } } peers { permanent_uuid: "e3eccc80d4134e2aaad313b4682f140f" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33651 } } peers { permanent_uuid: "86ef347370b84c4db0dd513e68ae51be" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 46311 } }
I20260927 10:57:17.508406 795 catalog_manager.cc:5495] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f reported cstate change: term changed from 0 to 1, leader changed from <none> to e3eccc80d4134e2aaad313b4682f140f (127.14.68.1). New cstate: current_term: 1 leader_uuid: "e3eccc80d4134e2aaad313b4682f140f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47a8f0c57a543438001b6d716492740" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3eccc80d4134e2aaad313b4682f140f" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 33651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86ef347370b84c4db0dd513e68ae51be" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 46311 } health_report { overall_health: UNKNOWN } } }
I20260927 10:57:17.531157 1259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:17.527793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56054 (local address 127.14.68.1:33651)
0927 10:57:17.527913 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:17.527922 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:17.527937 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:57:17.528043 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:17.528045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:17.528097 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:17.528182 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:57:17.528386 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.528878 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.529766 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:17.529963 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:17.530034 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:17.530334 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:57:17.530733 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:57:17.530840 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:17.531020 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 10:57:17.531051 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:57:17.536820 1199 raft_consensus.cc:1261] T 8d20f1cb73284f16947b6658bcab359b P e47a8f0c57a543438001b6d716492740 [term 1 FOLLOWER]: Refusing update from remote peer e3eccc80d4134e2aaad313b4682f140f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:17.536913 1068 raft_consensus.cc:1261] T 8d20f1cb73284f16947b6658bcab359b P 86ef347370b84c4db0dd513e68ae51be [term 1 FOLLOWER]: Refusing update from remote peer e3eccc80d4134e2aaad313b4682f140f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:17.537607 1269 consensus_queue.cc:1035] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47a8f0c57a543438001b6d716492740" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 42893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:17.537812 1273 consensus_queue.cc:1035] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f [LEADER]: Connected to new peer: Peer: permanent_uuid: "86ef347370b84c4db0dd513e68ae51be" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 46311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:17.553663 794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:57:17.553856 794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:57:17.556192 794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17f2deeed1cb4d5794ff46ac4b1fe1bd: Sending DeleteTablet for 3 replicas of tablet 8d20f1cb73284f16947b6658bcab359b
I20260927 10:57:17.556882 1179 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d20f1cb73284f16947b6658bcab359b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:17 UTC) from {username='slave'} at 127.0.0.1:52450
I20260927 10:57:17.556931 917 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d20f1cb73284f16947b6658bcab359b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:17 UTC) from {username='slave'} at 127.0.0.1:56052
I20260927 10:57:17.557313 1286 tablet_replica.cc:330] T 8d20f1cb73284f16947b6658bcab359b P e47a8f0c57a543438001b6d716492740: stopping tablet replica
I20260927 10:57:17.557358 1287 tablet_replica.cc:330] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f: stopping tablet replica
I20260927 10:57:17.557468 1048 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d20f1cb73284f16947b6658bcab359b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:17 UTC) from {username='slave'} at 127.0.0.1:51942
I20260927 10:57:17.557649 1287 raft_consensus.cc:2229] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:57:17.557752 1286 raft_consensus.cc:2229] T 8d20f1cb73284f16947b6658bcab359b P e47a8f0c57a543438001b6d716492740 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:57:17.558012 1287 raft_consensus.cc:2258] T 8d20f1cb73284f16947b6658bcab359b P e3eccc80d4134e2aaad313b4682f140f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:17.558051 1286 raft_consensus.cc:2258] T 8d20f1cb73284f16947b6658bcab359b P e47a8f0c57a543438001b6d716492740 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:17.558238 1288 tablet_replica.cc:330] T 8d20f1cb73284f16947b6658bcab359b P 86ef347370b84c4db0dd513e68ae51be: stopping tablet replica
I20260927 10:57:17.558516 1288 raft_consensus.cc:2229] T 8d20f1cb73284f16947b6658bcab359b P 86ef347370b84c4db0dd513e68ae51be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:57:17.558693 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 855
I20260927 10:57:17.559628 1288 raft_consensus.cc:2258] T 8d20f1cb73284f16947b6658bcab359b P 86ef347370b84c4db0dd513e68ae51be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:17.560599 1286 ts_tablet_manager.cc:1905] T 8d20f1cb73284f16947b6658bcab359b P e47a8f0c57a543438001b6d716492740: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:57:17.560956 1288 ts_tablet_manager.cc:1905] T 8d20f1cb73284f16947b6658bcab359b P 86ef347370b84c4db0dd513e68ae51be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:57:17.563350 1286 ts_tablet_manager.cc:1918] T 8d20f1cb73284f16947b6658bcab359b P e47a8f0c57a543438001b6d716492740: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:57:17.563465 1286 log.cc:1196] T 8d20f1cb73284f16947b6658bcab359b P e47a8f0c57a543438001b6d716492740: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/8d20f1cb73284f16947b6658bcab359b
I20260927 10:57:17.563555 1288 ts_tablet_manager.cc:1918] T 8d20f1cb73284f16947b6658bcab359b P 86ef347370b84c4db0dd513e68ae51be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:57:17.563673 1288 log.cc:1196] T 8d20f1cb73284f16947b6658bcab359b P 86ef347370b84c4db0dd513e68ae51be: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/8d20f1cb73284f16947b6658bcab359b
I20260927 10:57:17.563850 1286 ts_tablet_manager.cc:1939] T 8d20f1cb73284f16947b6658bcab359b P e47a8f0c57a543438001b6d716492740: Deleting consensus metadata
I20260927 10:57:17.564067 1288 ts_tablet_manager.cc:1939] T 8d20f1cb73284f16947b6658bcab359b P 86ef347370b84c4db0dd513e68ae51be: Deleting consensus metadata
I20260927 10:57:17.565209 781 catalog_manager.cc:4841] TS 86ef347370b84c4db0dd513e68ae51be (127.14.68.2:46311): tablet 8d20f1cb73284f16947b6658bcab359b (table test-table [id=571354aa3b074900a6da6e1d947a2ad1]) successfully deleted
I20260927 10:57:17.565459 781 catalog_manager.cc:4841] TS e47a8f0c57a543438001b6d716492740 (127.14.68.3:42893): tablet 8d20f1cb73284f16947b6658bcab359b (table test-table [id=571354aa3b074900a6da6e1d947a2ad1]) successfully deleted
W20260927 10:57:17.565653 781 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.68.1:33651 (error 108)
I20260927 10:57:17.565848 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 986
I20260927 10:57:17.567313 1260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:17.566948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:33651 (local address 127.0.0.1:56070)
0927 10:57:17.567137 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:17.567214 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.1:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260927 10:57:17.567481 781 catalog_manager.cc:4584] TS e3eccc80d4134e2aaad313b4682f140f (127.14.68.1:33651): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d20f1cb73284f16947b6658bcab359b: Network error: Client connection negotiation failed: client connection to 127.14.68.1:33651: connect: Connection refused (error 111)
I20260927 10:57:17.571571 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 1117
I20260927 10:57:17.576838 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 764
2026-09-27T10:57:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1333 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:17 dist-test-slave-tz51 krb5kdc[1292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:17 dist-test-slave-tz51 krb5kdc[1292](info): set up 2 sockets
Sep 27 10:57:17 dist-test-slave-tz51 krb5kdc[1292](info): commencing operation
krb5kdc: starting...
W20260927 10:57:18.615628 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:57:18 dist-test-slave-tz51 krb5kdc[1292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:57:18.734316 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:35973
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:35973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:18.842537 1305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:18.842815 1305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:18.842885 1305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:18.846843 1305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:18.846931 1305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:18.846971 1305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:18.847029 1305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:18.847065 1305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:18.852348 1305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:35973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:35973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:18.853338 1305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:18.854329 1305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:18.862620 1311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:18.862636 1310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:18.862673 1305 server_base.cc:1034] running on GCE node
W20260927 10:57:18.862622 1313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:18.863375 1305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:18.864429 1305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:18.865584 1305 hybrid_clock.cc:648] HybridClock initialized: now 1790506638865571 us; error 25 us; skew 500 ppm
Sep 27 10:57:18 dist-test-slave-tz51 krb5kdc[1292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506638, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:18.868224 1305 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:57:18.869038 1305 webserver.cc:460] Webserver started at http://127.14.68.62:40665/ using document root <none> and password file <none>
I20260927 10:57:18.869560 1305 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:18.869643 1305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:18.869865 1305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:18.871814 1305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "94c73250f235493cbd84fc4eaa05ed87"
format_stamp: "Formatted at 2026-09-27 10:57:18 on dist-test-slave-tz51"
server_key: "3b52f8a011ed2108b36c6d83ce43f2ba"
server_key_iv: "218baf3392042d29db7e42808f8fa25d"
server_key_version: "encryptionkey@0"
I20260927 10:57:18.872352 1305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "94c73250f235493cbd84fc4eaa05ed87"
format_stamp: "Formatted at 2026-09-27 10:57:18 on dist-test-slave-tz51"
server_key: "3b52f8a011ed2108b36c6d83ce43f2ba"
server_key_iv: "218baf3392042d29db7e42808f8fa25d"
server_key_version: "encryptionkey@0"
I20260927 10:57:18.875754 1305 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:57:18.877982 1320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:18.879117 1305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:18.879256 1305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "94c73250f235493cbd84fc4eaa05ed87"
format_stamp: "Formatted at 2026-09-27 10:57:18 on dist-test-slave-tz51"
server_key: "3b52f8a011ed2108b36c6d83ce43f2ba"
server_key_iv: "218baf3392042d29db7e42808f8fa25d"
server_key_version: "encryptionkey@0"
I20260927 10:57:18.879364 1305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:18.892798 1305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:18.895872 1305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:18.896070 1305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:18.902873 1305 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:35973
I20260927 10:57:18.902890 1372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:35973 every 8 connection(s)
I20260927 10:57:18.903817 1305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:18.906318 1373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:18.909904 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 1305
I20260927 10:57:18.910065 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:18.910405 14608 external_mini_cluster.cc:1428] Setting key 1178d28a3bc70b22994647a9e469d890
I20260927 10:57:18.911978 1373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87: Bootstrap starting.
I20260927 10:57:18.914022 1373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:18.914726 1373 log.cc:824] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:18.916465 1373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87: No bootstrap required, opened a new log
Sep 27 10:57:18 dist-test-slave-tz51 krb5kdc[1292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:57:18.920073 1373 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c73250f235493cbd84fc4eaa05ed87" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 35973 } }
I20260927 10:57:18.920272 1373 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:18.920346 1373 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94c73250f235493cbd84fc4eaa05ed87, State: Initialized, Role: FOLLOWER
I20260927 10:57:18.920754 1373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c73250f235493cbd84fc4eaa05ed87" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 35973 } }
I20260927 10:57:18.920892 1373 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:18.920974 1373 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:18.921084 1373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:18.922348 1373 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c73250f235493cbd84fc4eaa05ed87" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 35973 } }
I20260927 10:57:18.922686 1373 leader_election.cc:302] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94c73250f235493cbd84fc4eaa05ed87; no voters:
I20260927 10:57:18.923017 1373 leader_election.cc:288] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:18.923094 1378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:18.923339 1378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87 [term 1 LEADER]: Becoming Leader. State: Replica: 94c73250f235493cbd84fc4eaa05ed87, State: Running, Role: LEADER
I20260927 10:57:18.923674 1378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c73250f235493cbd84fc4eaa05ed87" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 35973 } }
I20260927 10:57:18.924274 1373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:18.925313 1380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94c73250f235493cbd84fc4eaa05ed87. Latest consensus state: current_term: 1 leader_uuid: "94c73250f235493cbd84fc4eaa05ed87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c73250f235493cbd84fc4eaa05ed87" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 35973 } } }
I20260927 10:57:18.925428 1380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:18.925546 1376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:18.911754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51946 (local address 127.14.68.62:35973)
0927 10:57:18.912193 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:18.912208 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:18.912243 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:57:18.912847 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:18.912862 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:18.913169 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:18.913487 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:18.914330 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:18.915363 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:18.916287 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:18.916788 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:18.916952 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:18.919611 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:18.919616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:18.919627 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:18.919669 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:18.921790 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:18.922299 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:18.922305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:18.922310 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:18.922486 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:18.922783 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:18.922786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:18.922788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:18.923305 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:18.923494 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:18.924286 (+ 792us) server_negotiation.cc:299] Negotiation successful
0927 10:57:18.924576 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":114,"threads_started":1}
I20260927 10:57:18.925706 1379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94c73250f235493cbd84fc4eaa05ed87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c73250f235493cbd84fc4eaa05ed87" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 35973 } } }
I20260927 10:57:18.925781 1379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:18.925809 1387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:18.928936 1387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:18.934340 1387 catalog_manager.cc:1347] Generated new cluster ID: ddd53fb4d0004c61bf2856d0ee004fae
I20260927 10:57:18.934412 1387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:18.952037 1387 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:18.953240 1387 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:18.958810 1387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87: Generated new TSK 0
I20260927 10:57:18.959543 1387 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.68.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
Entry for principal kudu/127.14.68.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab.
I20260927 10:57:18.997234 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:35973
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:19.104784 1399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:19.105057 1399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:19.105144 1399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:19.109086 1399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:19.109170 1399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:19.109256 1399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:57:19.114473 1399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.1.keytab
--principal=kudu/127.14.68.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:19.115444 1399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:19.116417 1399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:19.125798 1404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:19.125823 1407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:19.125873 1405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:19.126030 1399 server_base.cc:1034] running on GCE node
I20260927 10:57:19.126489 1399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:19.127141 1399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:19.128333 1399 hybrid_clock.cc:648] HybridClock initialized: now 1790506639128305 us; error 41 us; skew 500 ppm
Sep 27 10:57:19 dist-test-slave-tz51 krb5kdc[1292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506639, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:19.130976 1399 init.cc:377] Logged in from keytab as kudu/127.14.68.1@KRBTEST.COM (short username kudu)
I20260927 10:57:19.131850 1399 webserver.cc:460] Webserver started at http://127.14.68.1:39519/ using document root <none> and password file <none>
I20260927 10:57:19.132396 1399 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:19.132485 1399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:19.132755 1399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:19.134733 1399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "3f4ff641d34b432c92bba000c724bb94"
format_stamp: "Formatted at 2026-09-27 10:57:19 on dist-test-slave-tz51"
server_key: "4cd1a649142dbea918c68606cda188ee"
server_key_iv: "99b647469141d8dc5409789f556c82f6"
server_key_version: "encryptionkey@0"
I20260927 10:57:19.135279 1399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "3f4ff641d34b432c92bba000c724bb94"
format_stamp: "Formatted at 2026-09-27 10:57:19 on dist-test-slave-tz51"
server_key: "4cd1a649142dbea918c68606cda188ee"
server_key_iv: "99b647469141d8dc5409789f556c82f6"
server_key_version: "encryptionkey@0"
I20260927 10:57:19.138711 1399 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 10:57:19.141085 1414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:19.142204 1399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:57:19.142343 1399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "3f4ff641d34b432c92bba000c724bb94"
format_stamp: "Formatted at 2026-09-27 10:57:19 on dist-test-slave-tz51"
server_key: "4cd1a649142dbea918c68606cda188ee"
server_key_iv: "99b647469141d8dc5409789f556c82f6"
server_key_version: "encryptionkey@0"
I20260927 10:57:19.142446 1399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:19.165410 1399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:19.168174 1399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:19.168351 1399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:19.168932 1399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:19.169703 1399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:19.169749 1399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:19.169785 1399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:19.169807 1399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:19.177846 1399 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:37123
I20260927 10:57:19.177870 1527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:37123 every 8 connection(s)
I20260927 10:57:19.178639 1399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:57:19.183578 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 1399
I20260927 10:57:19.183750 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:57:19.184095 14608 external_mini_cluster.cc:1428] Setting key 66fb8c633e07948332ecac2ce78ba2c4
Sep 27 10:57:19 dist-test-slave-tz51 krb5kdc[1292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506639, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.1@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:57:19.192955 1376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:19.180341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:51609 (local address 127.14.68.62:35973)
0927 10:57:19.180465 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:19.180469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:19.181203 (+ 734us) server_negotiation.cc:407] Connection header received
0927 10:57:19.182569 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:19.182572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:19.182629 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:19.182730 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:19.184160 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.184852 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.186256 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.186475 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.186555 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:19.189318 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:19.189321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:19.189323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:19.189344 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:19.190792 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:19.191404 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:19.191407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:19.191408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:19.191479 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:19.191939 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:19.191941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:19.191943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:19.192101 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:19.192176 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:19.192749 (+ 573us) server_negotiation.cc:299] Negotiation successful
0927 10:57:19.192844 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:57:19.194064 1530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:19.180668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:35973 (local address 127.14.68.1:51609)
0927 10:57:19.181070 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:19.181107 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:19.182368 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:19.182901 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:19.182908 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:19.183231 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:19.183983 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:19.184005 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.184973 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.184976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:19.186131 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:19.186146 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.186616 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.186623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:19.186863 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:19.187508 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:19.187523 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:19.189160 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:19.190911 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:19.190920 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:19.190937 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:19.191307 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:19.191598 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:19.191604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:19.191608 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:19.191844 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:19.192208 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:19.192215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:19.192510 (+ 295us) client_negotiation.cc:769] Sending connection context
0927 10:57:19.192714 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 10:57:19.193042 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.14.68.2@KRBTEST.COM; defaulting to no policy
I20260927 10:57:19.195170 1528 heartbeater.cc:347] Connected to a master server at 127.14.68.62:35973
I20260927 10:57:19.195363 1528 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:19.195859 1528 heartbeater.cc:511] Master 127.14.68.62:35973 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.2@KRBTEST.COM" created.
I20260927 10:57:19.197367 1337 ts_manager.cc:194] Registered new tserver with Master: 3f4ff641d34b432c92bba000c724bb94 (127.14.68.1:37123)
I20260927 10:57:19.198434 1337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.1@KRBTEST.COM'} at 127.14.68.1:51609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
Entry for principal kudu/127.14.68.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab.
I20260927 10:57:19.212419 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:35973
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:19.317544 1533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:19.317800 1533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:19.317888 1533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:19.321826 1533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:19.321893 1533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:19.321974 1533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:57:19.327374 1533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.2.keytab
--principal=kudu/127.14.68.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:19.328274 1533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:19.329120 1533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:19.338016 1538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:19.338016 1541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:19.338037 1539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:19.338111 1533 server_base.cc:1034] running on GCE node
I20260927 10:57:19.338550 1533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:19.339152 1533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:19.340328 1533 hybrid_clock.cc:648] HybridClock initialized: now 1790506639340287 us; error 54 us; skew 500 ppm
Sep 27 10:57:19 dist-test-slave-tz51 krb5kdc[1292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506639, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:19.342864 1533 init.cc:377] Logged in from keytab as kudu/127.14.68.2@KRBTEST.COM (short username kudu)
I20260927 10:57:19.343681 1533 webserver.cc:460] Webserver started at http://127.14.68.2:43717/ using document root <none> and password file <none>
I20260927 10:57:19.344287 1533 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:19.344373 1533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:19.344587 1533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:19.346571 1533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "6a56d62b32ac4d7a8c7c9752e5399196"
format_stamp: "Formatted at 2026-09-27 10:57:19 on dist-test-slave-tz51"
server_key: "d56734246b5345ec62dc15104249103a"
server_key_iv: "831ad3ee1e4f59f4383ceeea5f9223a4"
server_key_version: "encryptionkey@0"
I20260927 10:57:19.347111 1533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "6a56d62b32ac4d7a8c7c9752e5399196"
format_stamp: "Formatted at 2026-09-27 10:57:19 on dist-test-slave-tz51"
server_key: "d56734246b5345ec62dc15104249103a"
server_key_iv: "831ad3ee1e4f59f4383ceeea5f9223a4"
server_key_version: "encryptionkey@0"
I20260927 10:57:19.350622 1533 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 10:57:19.352903 1548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:19.354002 1533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:57:19.354138 1533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "6a56d62b32ac4d7a8c7c9752e5399196"
format_stamp: "Formatted at 2026-09-27 10:57:19 on dist-test-slave-tz51"
server_key: "d56734246b5345ec62dc15104249103a"
server_key_iv: "831ad3ee1e4f59f4383ceeea5f9223a4"
server_key_version: "encryptionkey@0"
I20260927 10:57:19.354251 1533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:19.367198 1533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:19.369843 1533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:19.370028 1533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:19.370561 1533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:19.371330 1533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:19.371398 1533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:19.371469 1533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:19.371521 1533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:19.379897 1533 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:44477
I20260927 10:57:19.379922 1661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:44477 every 8 connection(s)
I20260927 10:57:19.380736 1533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:57:19.388007 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 1533
I20260927 10:57:19.388149 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:57:19.388521 14608 external_mini_cluster.cc:1428] Setting key ff4d1e0e41796fc648f63f3a68633a10
Sep 27 10:57:19 dist-test-slave-tz51 krb5kdc[1292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506639, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.2@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:57:19.394093 1376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:19.382312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:39901 (local address 127.14.68.62:35973)
0927 10:57:19.382458 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:19.382462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:19.383369 (+ 907us) server_negotiation.cc:407] Connection header received
0927 10:57:19.384736 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:19.384739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:19.384784 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:19.384856 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:19.385889 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.386423 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.387469 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.387675 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.387807 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:19.390448 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:19.390454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:19.390460 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:19.390490 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:19.392048 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:19.392634 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:19.392637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:19.392638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:19.392707 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:19.393069 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:19.393071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:19.393072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:19.393235 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:19.393307 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:19.393892 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 10:57:19.393983 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:57:19.395241 1664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:19.382763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:35973 (local address 127.14.68.2:39901)
0927 10:57:19.383231 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:19.383270 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:19.384556 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:19.385002 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:19.385009 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:19.385343 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:19.385773 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:19.385786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.386525 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.386528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:19.387371 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:19.387379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.387863 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.387867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:19.388078 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:19.388696 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:19.388715 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:19.390289 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:19.392153 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:19.392162 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:19.392177 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:19.392539 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:19.392796 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:19.392801 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:19.392805 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:19.392981 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:19.393337 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:19.393345 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:19.393656 (+ 311us) client_negotiation.cc:769] Sending connection context
0927 10:57:19.393861 (+ 205us) client_negotiation.cc:240] Negotiation successful
0927 10:57:19.394196 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
I20260927 10:57:19.396363 1662 heartbeater.cc:347] Connected to a master server at 127.14.68.62:35973
I20260927 10:57:19.396595 1662 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:19.397125 1662 heartbeater.cc:511] Master 127.14.68.62:35973 requested a full tablet report, sending...
I20260927 10:57:19.398296 1337 ts_manager.cc:194] Registered new tserver with Master: 6a56d62b32ac4d7a8c7c9752e5399196 (127.14.68.2:44477)
WARNING: no policy specified for kudu/127.14.68.3@KRBTEST.COM; defaulting to no policy
I20260927 10:57:19.399067 1337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.2@KRBTEST.COM'} at 127.14.68.2:39901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
Entry for principal kudu/127.14.68.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab.
I20260927 10:57:19.416337 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:35973
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:19.521898 1667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:19.522127 1667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:19.522200 1667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:19.526628 1667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:19.526684 1667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:19.526777 1667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:57:19.531821 1667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.3.keytab
--principal=kudu/127.14.68.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:19.532707 1667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:19.533615 1667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:19.542737 1672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:19.542867 1667 server_base.cc:1034] running on GCE node
W20260927 10:57:19.542783 1673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:19.543080 1675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:19.543426 1667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:19.544061 1667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:19.545214 1667 hybrid_clock.cc:648] HybridClock initialized: now 1790506639545201 us; error 25 us; skew 500 ppm
Sep 27 10:57:19 dist-test-slave-tz51 krb5kdc[1292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506639, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:19.547935 1667 init.cc:377] Logged in from keytab as kudu/127.14.68.3@KRBTEST.COM (short username kudu)
I20260927 10:57:19.548749 1667 webserver.cc:460] Webserver started at http://127.14.68.3:39363/ using document root <none> and password file <none>
I20260927 10:57:19.549269 1667 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:19.549326 1667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:19.549521 1667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:19.551386 1667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "6b97a733b4204e968f9759cef5fc0601"
format_stamp: "Formatted at 2026-09-27 10:57:19 on dist-test-slave-tz51"
server_key: "22176acb82a6503ea81617dea202454c"
server_key_iv: "9603efd92582a275f27347f40149970a"
server_key_version: "encryptionkey@0"
I20260927 10:57:19.551934 1667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "6b97a733b4204e968f9759cef5fc0601"
format_stamp: "Formatted at 2026-09-27 10:57:19 on dist-test-slave-tz51"
server_key: "22176acb82a6503ea81617dea202454c"
server_key_iv: "9603efd92582a275f27347f40149970a"
server_key_version: "encryptionkey@0"
I20260927 10:57:19.555234 1667 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:57:19.557593 1682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:19.558822 1667 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:19.558950 1667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "6b97a733b4204e968f9759cef5fc0601"
format_stamp: "Formatted at 2026-09-27 10:57:19 on dist-test-slave-tz51"
server_key: "22176acb82a6503ea81617dea202454c"
server_key_iv: "9603efd92582a275f27347f40149970a"
server_key_version: "encryptionkey@0"
I20260927 10:57:19.559055 1667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:19.572582 1667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:19.575227 1667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:19.575412 1667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:19.576012 1667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:19.576772 1667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:19.576841 1667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:19.576906 1667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:19.576956 1667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:19.585821 1667 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:33787
I20260927 10:57:19.585850 1795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:33787 every 8 connection(s)
I20260927 10:57:19.586668 1667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:57:19.591648 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 1667
I20260927 10:57:19.591787 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:57:19.592111 14608 external_mini_cluster.cc:1428] Setting key 083d40e1a88c7a14823c3df488286f66
Sep 27 10:57:19 dist-test-slave-tz51 krb5kdc[1292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506639, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.3@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:57:19.599846 1376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:19.588456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:44641 (local address 127.14.68.62:35973)
0927 10:57:19.588588 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:19.588591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:19.589299 (+ 708us) server_negotiation.cc:407] Connection header received
0927 10:57:19.590591 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:19.590594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:19.590652 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:19.590759 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:19.592191 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.592842 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.594293 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.594505 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.594585 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:19.596914 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:19.596917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:19.596919 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:19.596939 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:19.598143 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:19.598585 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:19.598588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:19.598589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:19.598659 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:19.598935 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:19.598937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:19.598938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:19.599095 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:19.599177 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:19.599578 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 10:57:19.599668 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:57:19.600559 1798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:19.588754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:35973 (local address 127.14.68.3:44641)
0927 10:57:19.589159 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:19.589198 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:19.590388 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:19.590920 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:19.590926 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:19.591306 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:19.591993 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:19.592014 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.592960 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.592963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:19.594171 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:19.594184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.594624 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.594627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:19.594761 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:19.595340 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:19.595355 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:19.596767 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:19.598257 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:19.598261 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:19.598272 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:19.598503 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:19.598740 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:19.598742 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:19.598743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:19.598860 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:19.599176 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:19.599180 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:19.599390 (+ 210us) client_negotiation.cc:769] Sending connection context
0927 10:57:19.599534 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 10:57:19.599813 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
I20260927 10:57:19.601537 1796 heartbeater.cc:347] Connected to a master server at 127.14.68.62:35973
I20260927 10:57:19.601730 1796 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:19.602123 1796 heartbeater.cc:511] Master 127.14.68.62:35973 requested a full tablet report, sending...
I20260927 10:57:19.602998 1337 ts_manager.cc:194] Registered new tserver with Master: 6b97a733b4204e968f9759cef5fc0601 (127.14.68.3:33787)
I20260927 10:57:19.603531 1337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.68.3@KRBTEST.COM'} at 127.14.68.3:44641
I20260927 10:57:19.605805 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:57:19.614846 1376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:19.608225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51954 (local address 127.14.68.62:35973)
0927 10:57:19.608424 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:19.608427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:19.608525 (+ 98us) server_negotiation.cc:407] Connection header received
0927 10:57:19.608631 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:19.608633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:19.608665 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:19.608731 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:19.609361 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.609819 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.610561 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.610750 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.610819 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:19.612098 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:19.612101 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:19.612102 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:19.612120 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:19.613493 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:19.613951 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:19.613953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:19.613954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:19.614018 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:19.614231 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:19.614233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:19.614234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:19.614407 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:19.614502 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:19.614667 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 10:57:19.614745 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260927 10:57:19.617534 1337 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:57:19.620458 1337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:57:19.634426 1808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:19.629149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:33787 (local address 127.0.0.1:48546)
0927 10:57:19.629843 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:19.629857 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:19.629936 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:19.630602 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:19.630606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:19.630622 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:19.630704 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:19.630711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.631787 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.631792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:19.632946 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:19.632956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.633863 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.633866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:19.633970 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:19.634010 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 10:57:19.634126 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 10:57:19.634268 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":63,"thread_start_us":89,"threads_started":1}
I20260927 10:57:19.634441 1812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:19.629744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.14.68.3:33787)
0927 10:57:19.630143 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:19.630175 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:19.630197 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:57:19.630263 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:19.630268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:19.630458 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:19.630626 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:19.630801 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.631587 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.633071 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.633738 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.633843 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:19.633978 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:19.634173 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 10:57:19.634269 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":132,"threads_started":1}
I20260927 10:57:19.634609 1810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:19.628534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:37123 (local address 127.0.0.1:58622)
0927 10:57:19.629442 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:19.629496 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:19.629613 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:19.630241 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:19.630247 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:19.630270 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:19.630386 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:19.630391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.632013 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.632016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:19.633284 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:19.633295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.634260 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.634263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:19.634343 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:19.634363 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:57:19.634408 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 10:57:19.634479 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":4,"thread_start_us":73,"threads_started":1}
I20260927 10:57:19.635258 1811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:19.629364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58622 (local address 127.14.68.1:37123)
0927 10:57:19.629714 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:19.629729 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:19.629751 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:57:19.629816 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:19.629820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:19.630025 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:19.630192 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:19.630510 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.631884 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.633438 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.634138 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.634790 (+ 652us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:19.634935 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:19.634982 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 10:57:19.635063 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":138,"threads_started":1}
I20260927 10:57:19.635299 1809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:19.628575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55592 (local address 127.14.68.2:44477)
0927 10:57:19.629194 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:19.629207 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:19.629579 (+ 372us) server_negotiation.cc:407] Connection header received
0927 10:57:19.629808 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:19.629814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:19.630018 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:19.630152 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:19.630636 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.631730 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.633519 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.634312 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.634411 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:19.634556 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:19.635043 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 10:57:19.635144 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":170,"threads_started":1}
I20260927 10:57:19.635664 1807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:19.628442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:44477 (local address 127.0.0.1:55592)
0927 10:57:19.629442 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:19.629492 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:19.629654 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:19.630369 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:19.630374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:19.630395 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:19.630508 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:19.630514 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.631881 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.631884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:19.633395 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:19.633404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.634621 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.634625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:19.634750 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:19.634775 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:57:19.635452 (+ 677us) client_negotiation.cc:240] Negotiation successful
0927 10:57:19.635533 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":57,"thread_start_us":68,"threads_started":1}
I20260927 10:57:19.637633 1730 tablet_service.cc:1467] Processing CreateTablet for tablet 45d7d4c561d647ada3e397977106f53a (DEFAULT_TABLE table=test-table [id=e3b438cddbbd44ea9f4842ac5b8b8aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:19.637676 1462 tablet_service.cc:1467] Processing CreateTablet for tablet 45d7d4c561d647ada3e397977106f53a (DEFAULT_TABLE table=test-table [id=e3b438cddbbd44ea9f4842ac5b8b8aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:19.638563 1462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d7d4c561d647ada3e397977106f53a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:19.638573 1730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d7d4c561d647ada3e397977106f53a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:19.644495 1813 tablet_bootstrap.cc:492] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94: Bootstrap starting.
I20260927 10:57:19.644315 1596 tablet_service.cc:1467] Processing CreateTablet for tablet 45d7d4c561d647ada3e397977106f53a (DEFAULT_TABLE table=test-table [id=e3b438cddbbd44ea9f4842ac5b8b8aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:19.645252 1596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d7d4c561d647ada3e397977106f53a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:19.645663 1814 tablet_bootstrap.cc:492] T 45d7d4c561d647ada3e397977106f53a P 6b97a733b4204e968f9759cef5fc0601: Bootstrap starting.
I20260927 10:57:19.646782 1813 tablet_bootstrap.cc:654] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:19.647742 1814 tablet_bootstrap.cc:654] T 45d7d4c561d647ada3e397977106f53a P 6b97a733b4204e968f9759cef5fc0601: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:19.648154 1813 log.cc:824] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:19.648969 1814 log.cc:824] T 45d7d4c561d647ada3e397977106f53a P 6b97a733b4204e968f9759cef5fc0601: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:19.650601 1813 tablet_bootstrap.cc:492] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94: No bootstrap required, opened a new log
I20260927 10:57:19.650776 1813 ts_tablet_manager.cc:1397] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:57:19.650919 1814 tablet_bootstrap.cc:492] T 45d7d4c561d647ada3e397977106f53a P 6b97a733b4204e968f9759cef5fc0601: No bootstrap required, opened a new log
I20260927 10:57:19.651114 1814 ts_tablet_manager.cc:1397] T 45d7d4c561d647ada3e397977106f53a P 6b97a733b4204e968f9759cef5fc0601: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:57:19.651573 1817 tablet_bootstrap.cc:492] T 45d7d4c561d647ada3e397977106f53a P 6a56d62b32ac4d7a8c7c9752e5399196: Bootstrap starting.
I20260927 10:57:19.653679 1817 tablet_bootstrap.cc:654] T 45d7d4c561d647ada3e397977106f53a P 6a56d62b32ac4d7a8c7c9752e5399196: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:19.653905 1813 raft_consensus.cc:348] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a56d62b32ac4d7a8c7c9752e5399196" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 44477 } } peers { permanent_uuid: "3f4ff641d34b432c92bba000c724bb94" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37123 } } peers { permanent_uuid: "6b97a733b4204e968f9759cef5fc0601" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 33787 } }
I20260927 10:57:19.654117 1813 raft_consensus.cc:374] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:19.654196 1813 raft_consensus.cc:729] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f4ff641d34b432c92bba000c724bb94, State: Initialized, Role: FOLLOWER
I20260927 10:57:19.654619 1813 consensus_queue.cc:260] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a56d62b32ac4d7a8c7c9752e5399196" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 44477 } } peers { permanent_uuid: "3f4ff641d34b432c92bba000c724bb94" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37123 } } peers { permanent_uuid: "6b97a733b4204e968f9759cef5fc0601" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 33787 } }
I20260927 10:57:19.654824 1817 log.cc:824] T 45d7d4c561d647ada3e397977106f53a P 6a56d62b32ac4d7a8c7c9752e5399196: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:19.654877 1814 raft_consensus.cc:348] T 45d7d4c561d647ada3e397977106f53a P 6b97a733b4204e968f9759cef5fc0601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a56d62b32ac4d7a8c7c9752e5399196" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 44477 } } peers { permanent_uuid: "3f4ff641d34b432c92bba000c724bb94" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37123 } } peers { permanent_uuid: "6b97a733b4204e968f9759cef5fc0601" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 33787 } }
I20260927 10:57:19.655157 1814 raft_consensus.cc:374] T 45d7d4c561d647ada3e397977106f53a P 6b97a733b4204e968f9759cef5fc0601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:19.655215 1528 heartbeater.cc:503] Master 127.14.68.62:35973 was elected leader, sending a full tablet report...
I20260927 10:57:19.655267 1814 raft_consensus.cc:729] T 45d7d4c561d647ada3e397977106f53a P 6b97a733b4204e968f9759cef5fc0601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b97a733b4204e968f9759cef5fc0601, State: Initialized, Role: FOLLOWER
I20260927 10:57:19.655689 1814 consensus_queue.cc:260] T 45d7d4c561d647ada3e397977106f53a P 6b97a733b4204e968f9759cef5fc0601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a56d62b32ac4d7a8c7c9752e5399196" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 44477 } } peers { permanent_uuid: "3f4ff641d34b432c92bba000c724bb94" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37123 } } peers { permanent_uuid: "6b97a733b4204e968f9759cef5fc0601" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 33787 } }
I20260927 10:57:19.656281 1796 heartbeater.cc:503] Master 127.14.68.62:35973 was elected leader, sending a full tablet report...
I20260927 10:57:19.656641 1814 ts_tablet_manager.cc:1428] T 45d7d4c561d647ada3e397977106f53a P 6b97a733b4204e968f9759cef5fc0601: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:57:19.657546 1817 tablet_bootstrap.cc:492] T 45d7d4c561d647ada3e397977106f53a P 6a56d62b32ac4d7a8c7c9752e5399196: No bootstrap required, opened a new log
I20260927 10:57:19.657742 1817 ts_tablet_manager.cc:1397] T 45d7d4c561d647ada3e397977106f53a P 6a56d62b32ac4d7a8c7c9752e5399196: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260927 10:57:19.658568 1813 ts_tablet_manager.cc:1428] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260927 10:57:19.661087 1817 raft_consensus.cc:348] T 45d7d4c561d647ada3e397977106f53a P 6a56d62b32ac4d7a8c7c9752e5399196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a56d62b32ac4d7a8c7c9752e5399196" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 44477 } } peers { permanent_uuid: "3f4ff641d34b432c92bba000c724bb94" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37123 } } peers { permanent_uuid: "6b97a733b4204e968f9759cef5fc0601" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 33787 } }
I20260927 10:57:19.661298 1817 raft_consensus.cc:374] T 45d7d4c561d647ada3e397977106f53a P 6a56d62b32ac4d7a8c7c9752e5399196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:19.661381 1817 raft_consensus.cc:729] T 45d7d4c561d647ada3e397977106f53a P 6a56d62b32ac4d7a8c7c9752e5399196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a56d62b32ac4d7a8c7c9752e5399196, State: Initialized, Role: FOLLOWER
I20260927 10:57:19.661731 1817 consensus_queue.cc:260] T 45d7d4c561d647ada3e397977106f53a P 6a56d62b32ac4d7a8c7c9752e5399196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a56d62b32ac4d7a8c7c9752e5399196" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 44477 } } peers { permanent_uuid: "3f4ff641d34b432c92bba000c724bb94" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37123 } } peers { permanent_uuid: "6b97a733b4204e968f9759cef5fc0601" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 33787 } }
I20260927 10:57:19.662287 1662 heartbeater.cc:503] Master 127.14.68.62:35973 was elected leader, sending a full tablet report...
I20260927 10:57:19.662525 1817 ts_tablet_manager.cc:1428] T 45d7d4c561d647ada3e397977106f53a P 6a56d62b32ac4d7a8c7c9752e5399196: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 10:57:19.679638 1529 tablet.cc:2369] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:57:19.837754 1797 tablet.cc:2369] T 45d7d4c561d647ada3e397977106f53a P 6b97a733b4204e968f9759cef5fc0601: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:57:19.881807 1663 tablet.cc:2369] T 45d7d4c561d647ada3e397977106f53a P 6a56d62b32ac4d7a8c7c9752e5399196: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:57:19.929252 1818 raft_consensus.cc:482] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:57:19.929469 1818 raft_consensus.cc:504] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a56d62b32ac4d7a8c7c9752e5399196" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 44477 } } peers { permanent_uuid: "3f4ff641d34b432c92bba000c724bb94" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37123 } } peers { permanent_uuid: "6b97a733b4204e968f9759cef5fc0601" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 33787 } }
I20260927 10:57:19.930763 1818 leader_election.cc:288] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a56d62b32ac4d7a8c7c9752e5399196 (127.14.68.2:44477), 6b97a733b4204e968f9759cef5fc0601 (127.14.68.3:33787)
I20260927 10:57:19.934793 1822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:19.930983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:44477 (local address 127.14.68.1:39105)
0927 10:57:19.931295 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:19.931312 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:19.931461 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:19.931825 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:19.931828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:19.931864 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:19.932008 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:19.932015 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.932829 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.932832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:19.933733 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:19.933741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.934368 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.934373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:19.934461 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:19.934488 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:57:19.934560 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 10:57:19.934613 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"spinlock_wait_cycles":1792,"thread_start_us":111,"threads_started":1}
I20260927 10:57:19.934792 1809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:19.931146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:39105 (local address 127.14.68.2:44477)
0927 10:57:19.931294 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:19.931299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:19.931400 (+ 101us) server_negotiation.cc:407] Connection header received
0927 10:57:19.931601 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:19.931604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:19.931665 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:19.931772 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:19.932130 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.932712 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.933856 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.934248 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.934324 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:19.934398 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:19.934587 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 10:57:19.934676 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:57:19.934870 1812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:19.931560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:50289 (local address 127.14.68.3:33787)
0927 10:57:19.931759 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:19.931768 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:19.931869 (+ 101us) server_negotiation.cc:407] Connection header received
0927 10:57:19.931974 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:19.931977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:19.932026 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:19.932100 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:19.932348 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.932873 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.933916 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.934311 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.934411 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:19.934468 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:19.934690 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 10:57:19.934751 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:57:19.935039 1823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:19.931407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:33787 (local address 127.14.68.1:50289)
0927 10:57:19.931786 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:19.931799 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:19.931888 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:19.932125 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:19.932127 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:19.932144 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:19.932248 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:19.932254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.932979 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.932982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:19.933777 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:19.933807 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.934432 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.934435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:19.934561 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:19.934586 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:57:19.934658 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 10:57:19.934720 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":132,"threads_started":1}
I20260927 10:57:19.935601 1616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d7d4c561d647ada3e397977106f53a" candidate_uuid: "3f4ff641d34b432c92bba000c724bb94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a56d62b32ac4d7a8c7c9752e5399196" is_pre_election: true
I20260927 10:57:19.935650 1750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d7d4c561d647ada3e397977106f53a" candidate_uuid: "3f4ff641d34b432c92bba000c724bb94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b97a733b4204e968f9759cef5fc0601" is_pre_election: true
I20260927 10:57:19.935949 1616 raft_consensus.cc:2454] T 45d7d4c561d647ada3e397977106f53a P 6a56d62b32ac4d7a8c7c9752e5399196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f4ff641d34b432c92bba000c724bb94 in term 0.
I20260927 10:57:19.936007 1750 raft_consensus.cc:2454] T 45d7d4c561d647ada3e397977106f53a P 6b97a733b4204e968f9759cef5fc0601 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f4ff641d34b432c92bba000c724bb94 in term 0.
I20260927 10:57:19.936450 1415 leader_election.cc:302] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f4ff641d34b432c92bba000c724bb94, 6a56d62b32ac4d7a8c7c9752e5399196; no voters:
I20260927 10:57:19.936718 1818 raft_consensus.cc:2781] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:57:19.936810 1818 raft_consensus.cc:482] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:57:19.936872 1818 raft_consensus.cc:3037] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:19.937855 1818 raft_consensus.cc:504] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a56d62b32ac4d7a8c7c9752e5399196" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 44477 } } peers { permanent_uuid: "3f4ff641d34b432c92bba000c724bb94" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37123 } } peers { permanent_uuid: "6b97a733b4204e968f9759cef5fc0601" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 33787 } }
I20260927 10:57:19.938189 1818 leader_election.cc:288] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94 [CANDIDATE]: Term 1 election: Requested vote from peers 6a56d62b32ac4d7a8c7c9752e5399196 (127.14.68.2:44477), 6b97a733b4204e968f9759cef5fc0601 (127.14.68.3:33787)
I20260927 10:57:19.938524 1616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d7d4c561d647ada3e397977106f53a" candidate_uuid: "3f4ff641d34b432c92bba000c724bb94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a56d62b32ac4d7a8c7c9752e5399196"
I20260927 10:57:19.938627 1750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d7d4c561d647ada3e397977106f53a" candidate_uuid: "3f4ff641d34b432c92bba000c724bb94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b97a733b4204e968f9759cef5fc0601"
I20260927 10:57:19.938702 1616 raft_consensus.cc:3037] T 45d7d4c561d647ada3e397977106f53a P 6a56d62b32ac4d7a8c7c9752e5399196 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:19.938737 1750 raft_consensus.cc:3037] T 45d7d4c561d647ada3e397977106f53a P 6b97a733b4204e968f9759cef5fc0601 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:19.939858 1616 raft_consensus.cc:2454] T 45d7d4c561d647ada3e397977106f53a P 6a56d62b32ac4d7a8c7c9752e5399196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f4ff641d34b432c92bba000c724bb94 in term 1.
I20260927 10:57:19.939859 1750 raft_consensus.cc:2454] T 45d7d4c561d647ada3e397977106f53a P 6b97a733b4204e968f9759cef5fc0601 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f4ff641d34b432c92bba000c724bb94 in term 1.
I20260927 10:57:19.940146 1415 leader_election.cc:302] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f4ff641d34b432c92bba000c724bb94, 6a56d62b32ac4d7a8c7c9752e5399196; no voters:
I20260927 10:57:19.940340 1818 raft_consensus.cc:2781] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:19.940563 1818 raft_consensus.cc:686] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94 [term 1 LEADER]: Becoming Leader. State: Replica: 3f4ff641d34b432c92bba000c724bb94, State: Running, Role: LEADER
I20260927 10:57:19.940860 1818 consensus_queue.cc:237] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a56d62b32ac4d7a8c7c9752e5399196" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 44477 } } peers { permanent_uuid: "3f4ff641d34b432c92bba000c724bb94" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37123 } } peers { permanent_uuid: "6b97a733b4204e968f9759cef5fc0601" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 33787 } }
I20260927 10:57:19.943843 1337 catalog_manager.cc:5495] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f4ff641d34b432c92bba000c724bb94 (127.14.68.1). New cstate: current_term: 1 leader_uuid: "3f4ff641d34b432c92bba000c724bb94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a56d62b32ac4d7a8c7c9752e5399196" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 44477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f4ff641d34b432c92bba000c724bb94" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 37123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b97a733b4204e968f9759cef5fc0601" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 33787 } health_report { overall_health: UNKNOWN } } }
I20260927 10:57:19.993568 1811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:19.990357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58632 (local address 127.14.68.1:37123)
0927 10:57:19.990490 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:19.990494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:19.990508 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:57:19.990641 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:19.990643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:19.990697 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:19.990785 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:57:19.990996 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.991496 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.992299 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:19.992497 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:19.992568 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:19.992811 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:57:19.993210 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:57:19.993297 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:19.993350 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 10:57:19.993394 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:57:19.999047 1750 raft_consensus.cc:1261] T 45d7d4c561d647ada3e397977106f53a P 6b97a733b4204e968f9759cef5fc0601 [term 1 FOLLOWER]: Refusing update from remote peer 3f4ff641d34b432c92bba000c724bb94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:19.999073 1616 raft_consensus.cc:1261] T 45d7d4c561d647ada3e397977106f53a P 6a56d62b32ac4d7a8c7c9752e5399196 [term 1 FOLLOWER]: Refusing update from remote peer 3f4ff641d34b432c92bba000c724bb94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:19.999760 1824 consensus_queue.cc:1035] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b97a733b4204e968f9759cef5fc0601" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 33787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:19.999917 1818 consensus_queue.cc:1035] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a56d62b32ac4d7a8c7c9752e5399196" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 44477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:20.009115 1828 mvcc.cc:205] Tried to move back new op lower bound from 7333915197429395456 to 7333915197200306176. Current Snapshot: MvccSnapshot[applied={T|T < 7333915197429395456}]
I20260927 10:57:20.015141 1335 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:57:20.015280 1335 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:57:20.017479 1335 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94c73250f235493cbd84fc4eaa05ed87: Sending DeleteTablet for 3 replicas of tablet 45d7d4c561d647ada3e397977106f53a
I20260927 10:57:20.018222 1596 tablet_service.cc:1514] Processing DeleteTablet for tablet 45d7d4c561d647ada3e397977106f53a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:20 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:55592
I20260927 10:57:20.018254 1730 tablet_service.cc:1514] Processing DeleteTablet for tablet 45d7d4c561d647ada3e397977106f53a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:20 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:48546
I20260927 10:57:20.018249 1462 tablet_service.cc:1514] Processing DeleteTablet for tablet 45d7d4c561d647ada3e397977106f53a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:20 UTC) from {username='kudu', principal='kudu/127.14.68.62@KRBTEST.COM'} at 127.0.0.1:58622
I20260927 10:57:20.018680 1839 tablet_replica.cc:330] T 45d7d4c561d647ada3e397977106f53a P 6a56d62b32ac4d7a8c7c9752e5399196: stopping tablet replica
I20260927 10:57:20.018852 1838 tablet_replica.cc:330] T 45d7d4c561d647ada3e397977106f53a P 3f4ff641d34b432c92bba000c724bb94: stopping tablet replica
I20260927 10:57:20.019024 1839 raft_consensus.cc:2229] T 45d7d4c561d647ada3e397977106f53a P 6a56d62b32ac4d7a8c7c9752e5399196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:57:20.019264 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 1399
I20260927 10:57:20.019372 1839 raft_consensus.cc:2258] T 45d7d4c561d647ada3e397977106f53a P 6a56d62b32ac4d7a8c7c9752e5399196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:20.020962 1837 tablet_replica.cc:330] T 45d7d4c561d647ada3e397977106f53a P 6b97a733b4204e968f9759cef5fc0601: stopping tablet replica
I20260927 10:57:20.021236 1837 raft_consensus.cc:2229] T 45d7d4c561d647ada3e397977106f53a P 6b97a733b4204e968f9759cef5fc0601 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:57:20.021306 1839 ts_tablet_manager.cc:1905] T 45d7d4c561d647ada3e397977106f53a P 6a56d62b32ac4d7a8c7c9752e5399196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:57:20.021512 1837 raft_consensus.cc:2258] T 45d7d4c561d647ada3e397977106f53a P 6b97a733b4204e968f9759cef5fc0601 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:20.022547 1837 ts_tablet_manager.cc:1905] T 45d7d4c561d647ada3e397977106f53a P 6b97a733b4204e968f9759cef5fc0601: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:57:20.024600 1837 ts_tablet_manager.cc:1918] T 45d7d4c561d647ada3e397977106f53a P 6b97a733b4204e968f9759cef5fc0601: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:57:20.024703 1837 log.cc:1196] T 45d7d4c561d647ada3e397977106f53a P 6b97a733b4204e968f9759cef5fc0601: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/45d7d4c561d647ada3e397977106f53a
I20260927 10:57:20.024893 1839 ts_tablet_manager.cc:1918] T 45d7d4c561d647ada3e397977106f53a P 6a56d62b32ac4d7a8c7c9752e5399196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:57:20.025012 1839 log.cc:1196] T 45d7d4c561d647ada3e397977106f53a P 6a56d62b32ac4d7a8c7c9752e5399196: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/45d7d4c561d647ada3e397977106f53a
I20260927 10:57:20.025146 1837 ts_tablet_manager.cc:1939] T 45d7d4c561d647ada3e397977106f53a P 6b97a733b4204e968f9759cef5fc0601: Deleting consensus metadata
I20260927 10:57:20.025346 1839 ts_tablet_manager.cc:1939] T 45d7d4c561d647ada3e397977106f53a P 6a56d62b32ac4d7a8c7c9752e5399196: Deleting consensus metadata
I20260927 10:57:20.026271 1321 catalog_manager.cc:4841] TS 6b97a733b4204e968f9759cef5fc0601 (127.14.68.3:33787): tablet 45d7d4c561d647ada3e397977106f53a (table test-table [id=e3b438cddbbd44ea9f4842ac5b8b8aa5]) successfully deleted
I20260927 10:57:20.026461 1321 catalog_manager.cc:4841] TS 6a56d62b32ac4d7a8c7c9752e5399196 (127.14.68.2:44477): tablet 45d7d4c561d647ada3e397977106f53a (table test-table [id=e3b438cddbbd44ea9f4842ac5b8b8aa5]) successfully deleted
W20260927 10:57:20.027086 1324 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.68.1:37123 (error 108)
I20260927 10:57:20.027793 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 1533
I20260927 10:57:20.028070 1807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:20.027790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:37123 (local address 127.0.0.1:58640)
0927 10:57:20.027908 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:20.027984 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.1:37123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 10:57:20.031858 1324 catalog_manager.cc:4584] TS 3f4ff641d34b432c92bba000c724bb94 (127.14.68.1:37123): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 45d7d4c561d647ada3e397977106f53a: Network error: Client connection negotiation failed: client connection to 127.14.68.1:37123: connect: Connection refused (error 111)
I20260927 10:57:20.033272 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 1667
I20260927 10:57:20.038820 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 1305
2026-09-27T10:57:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2463 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T10:57:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:20Z Disabled control of system clock
I20260927 10:57:20.068522 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:46425
--webserver_interface=127.14.68.62
--webserver_port=0
--builtin_ntp_servers=127.14.68.20:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:46425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:57:20.173799 1844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:20.174034 1844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:20.174068 1844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:20.177968 1844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:20.178020 1844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:20.178036 1844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:20.178052 1844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:20.178066 1844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:20.182720 1844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:46425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:46425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:20.183723 1844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:20.185024 1844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:20.192864 1850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:20.192873 1849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:20.193017 1844 server_base.cc:1034] running on GCE node
W20260927 10:57:20.193173 1852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:20.193650 1844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:20.194624 1844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:20.195765 1844 hybrid_clock.cc:648] HybridClock initialized: now 1790506640195746 us; error 29 us; skew 500 ppm
I20260927 10:57:20.197558 1844 webserver.cc:460] Webserver started at http://127.14.68.62:43903/ using document root <none> and password file <none>
I20260927 10:57:20.198124 1844 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:20.198189 1844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:20.198410 1844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:20.200368 1844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "6cc2d1723d324a74a8516c9d3c239a03"
format_stamp: "Formatted at 2026-09-27 10:57:20 on dist-test-slave-tz51"
server_key: "8d8457ef423ed60eab3b562d20874201"
server_key_iv: "41a46dfa1b842831e65a2227e3b48b6d"
server_key_version: "encryptionkey@0"
I20260927 10:57:20.200901 1844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "6cc2d1723d324a74a8516c9d3c239a03"
format_stamp: "Formatted at 2026-09-27 10:57:20 on dist-test-slave-tz51"
server_key: "8d8457ef423ed60eab3b562d20874201"
server_key_iv: "41a46dfa1b842831e65a2227e3b48b6d"
server_key_version: "encryptionkey@0"
I20260927 10:57:20.204226 1844 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:57:20.206423 1858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:20.207540 1844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:57:20.207671 1844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "6cc2d1723d324a74a8516c9d3c239a03"
format_stamp: "Formatted at 2026-09-27 10:57:20 on dist-test-slave-tz51"
server_key: "8d8457ef423ed60eab3b562d20874201"
server_key_iv: "41a46dfa1b842831e65a2227e3b48b6d"
server_key_version: "encryptionkey@0"
I20260927 10:57:20.207826 1844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:20.228587 1844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:20.229310 1844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:20.229511 1844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:20.235924 1844 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:46425
I20260927 10:57:20.235936 1910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:46425 every 8 connection(s)
I20260927 10:57:20.236848 1844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:20.239923 1911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:20.243623 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 1844
I20260927 10:57:20.243778 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:20.244107 14608 external_mini_cluster.cc:1428] Setting key a7ae7dc56814fc2481117c070aad682b
I20260927 10:57:20.245878 1911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03: Bootstrap starting.
I20260927 10:57:20.248063 1911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:20.248776 1911 log.cc:824] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:20.250537 1911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03: No bootstrap required, opened a new log
I20260927 10:57:20.253142 1914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:20.245574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57396 (local address 127.14.68.62:46425)
0927 10:57:20.246235 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:20.246251 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:20.246285 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:57:20.246906 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:20.246922 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:20.247306 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:20.247632 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:20.248079 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.249128 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.250030 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.250490 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.250662 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:20.251527 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:20.251532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:20.251544 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:20.251977 (+ 433us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:20.252029 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:20.252048 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:20.252155 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:20.252266 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 10:57:20.252499 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":122,"threads_started":1}
I20260927 10:57:20.254251 1911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc2d1723d324a74a8516c9d3c239a03" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46425 } }
I20260927 10:57:20.254469 1911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:20.254544 1911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc2d1723d324a74a8516c9d3c239a03, State: Initialized, Role: FOLLOWER
I20260927 10:57:20.254992 1911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc2d1723d324a74a8516c9d3c239a03" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46425 } }
I20260927 10:57:20.255141 1911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:20.255216 1911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:20.255311 1911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:20.256184 1911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc2d1723d324a74a8516c9d3c239a03" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46425 } }
I20260927 10:57:20.256455 1911 leader_election.cc:302] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cc2d1723d324a74a8516c9d3c239a03; no voters:
I20260927 10:57:20.256669 1911 leader_election.cc:288] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:20.256791 1916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:20.257011 1916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03 [term 1 LEADER]: Becoming Leader. State: Replica: 6cc2d1723d324a74a8516c9d3c239a03, State: Running, Role: LEADER
I20260927 10:57:20.257344 1916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc2d1723d324a74a8516c9d3c239a03" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46425 } }
I20260927 10:57:20.257764 1911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:20.258951 1917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cc2d1723d324a74a8516c9d3c239a03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc2d1723d324a74a8516c9d3c239a03" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46425 } } }
I20260927 10:57:20.258944 1918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cc2d1723d324a74a8516c9d3c239a03. Latest consensus state: current_term: 1 leader_uuid: "6cc2d1723d324a74a8516c9d3c239a03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc2d1723d324a74a8516c9d3c239a03" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 46425 } } }
I20260927 10:57:20.259085 1917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:20.259089 1918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:20.260154 1925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:20.263082 1925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:20.268729 1925 catalog_manager.cc:1347] Generated new cluster ID: 133aee9e93f74a85a5d3feff82480440
I20260927 10:57:20.268802 1925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:20.285526 1925 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:20.286427 1925 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:20.292444 1925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03: Generated new TSK 0
I20260927 10:57:20.293243 1925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:57:20.296109 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.1:0
--local_ip_for_outbound_sockets=127.14.68.1
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46425
--builtin_ntp_servers=127.14.68.20:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:57:20.405712 1935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:20.405947 1935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:20.405983 1935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:20.409984 1935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:20.410045 1935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:20.410100 1935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.1
I20260927 10:57:20.414923 1935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.68.1
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.1
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:20.415938 1935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:20.417204 1935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:20.426321 1943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:20.426347 1940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:20.426447 1941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:20.426488 1935 server_base.cc:1034] running on GCE node
I20260927 10:57:20.427006 1935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:20.427580 1935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:20.428731 1935 hybrid_clock.cc:648] HybridClock initialized: now 1790506640428718 us; error 27 us; skew 500 ppm
I20260927 10:57:20.430384 1935 webserver.cc:460] Webserver started at http://127.14.68.1:38341/ using document root <none> and password file <none>
I20260927 10:57:20.430909 1935 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:20.430975 1935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:20.431208 1935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:20.433138 1935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-0/data/instance:
uuid: "ed55a26ff72f4a06a8a24a721f5de9a5"
format_stamp: "Formatted at 2026-09-27 10:57:20 on dist-test-slave-tz51"
server_key: "b2655ccdbe72e3079bd8bb956923249d"
server_key_iv: "75eec0bc546d53026478266b809b987a"
server_key_version: "encryptionkey@0"
I20260927 10:57:20.433678 1935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance:
uuid: "ed55a26ff72f4a06a8a24a721f5de9a5"
format_stamp: "Formatted at 2026-09-27 10:57:20 on dist-test-slave-tz51"
server_key: "b2655ccdbe72e3079bd8bb956923249d"
server_key_iv: "75eec0bc546d53026478266b809b987a"
server_key_version: "encryptionkey@0"
I20260927 10:57:20.436998 1935 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:57:20.439275 1949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:20.440466 1935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:20.440584 1935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-0/wal
uuid: "ed55a26ff72f4a06a8a24a721f5de9a5"
format_stamp: "Formatted at 2026-09-27 10:57:20 on dist-test-slave-tz51"
server_key: "b2655ccdbe72e3079bd8bb956923249d"
server_key_iv: "75eec0bc546d53026478266b809b987a"
server_key_version: "encryptionkey@0"
I20260927 10:57:20.440685 1935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:20.451416 1935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:20.452034 1935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:20.452207 1935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:20.452696 1935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:20.453505 1935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:20.453569 1935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:20.453627 1935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:20.453675 1935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:20.462381 1935 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.1:41261
I20260927 10:57:20.462402 2062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.1:41261 every 8 connection(s)
I20260927 10:57:20.463199 1935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-0/data/info.pb
I20260927 10:57:20.471992 1914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:20.464898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.1:35387 (local address 127.14.68.62:46425)
0927 10:57:20.465022 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:20.465026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:20.465990 (+ 964us) server_negotiation.cc:407] Connection header received
0927 10:57:20.467411 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:20.467414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:20.467478 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:20.467581 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:20.468455 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.468988 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.470064 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.470300 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.470412 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:20.471446 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:20.471451 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:20.471458 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:20.471469 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:20.471485 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:20.471492 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:20.471565 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:20.471794 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 10:57:20.471854 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:57:20.472131 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 1935
I20260927 10:57:20.472276 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-0/wal/instance
I20260927 10:57:20.472555 14608 external_mini_cluster.cc:1428] Setting key 984f76e79458c92db1f291bf43090eb7
I20260927 10:57:20.472909 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:20.465272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:46425 (local address 127.14.68.1:35387)
0927 10:57:20.465856 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:20.465892 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:20.467157 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:20.467787 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:20.467794 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:20.467873 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:20.468316 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:20.468327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.469115 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.469118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:20.469941 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:20.469950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.470448 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.470452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:20.470605 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:20.471214 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:57:20.471216 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:20.471252 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:57:20.471306 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:20.471581 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:20.471595 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:20.471601 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:57:20.471792 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 10:57:20.472043 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":103,"threads_started":1}
I20260927 10:57:20.474044 2063 heartbeater.cc:347] Connected to a master server at 127.14.68.62:46425
I20260927 10:57:20.474277 2063 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:20.474553 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.2:0
--local_ip_for_outbound_sockets=127.14.68.2
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46425
--builtin_ntp_servers=127.14.68.20:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:57:20.474839 2063 heartbeater.cc:511] Master 127.14.68.62:46425 requested a full tablet report, sending...
I20260927 10:57:20.476363 1875 ts_manager.cc:194] Registered new tserver with Master: ed55a26ff72f4a06a8a24a721f5de9a5 (127.14.68.1:41261)
I20260927 10:57:20.477409 1875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.68.1:35387
W20260927 10:57:20.584441 2066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:20.584663 2066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:20.584697 2066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:20.588581 2066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:20.588620 2066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:20.588670 2066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.2
I20260927 10:57:20.593106 2066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.68.2
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.2
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:20.594008 2066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:20.595232 2066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:20.604138 2072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:20.604131 2074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:20.604251 2066 server_base.cc:1034] running on GCE node
W20260927 10:57:20.604131 2071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:20.604753 2066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:20.605360 2066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:20.606539 2066 hybrid_clock.cc:648] HybridClock initialized: now 1790506640606526 us; error 25 us; skew 500 ppm
I20260927 10:57:20.608256 2066 webserver.cc:460] Webserver started at http://127.14.68.2:44469/ using document root <none> and password file <none>
I20260927 10:57:20.608806 2066 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:20.608902 2066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:20.609114 2066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:20.611140 2066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-1/data/instance:
uuid: "5279f6b42e0a486baa534254db2bd8c7"
format_stamp: "Formatted at 2026-09-27 10:57:20 on dist-test-slave-tz51"
server_key: "1b56b9befda35e7cf8884462b58eb9d2"
server_key_iv: "c4f05d5d3ad38a7ae24af6926d76422b"
server_key_version: "encryptionkey@0"
I20260927 10:57:20.611812 2066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance:
uuid: "5279f6b42e0a486baa534254db2bd8c7"
format_stamp: "Formatted at 2026-09-27 10:57:20 on dist-test-slave-tz51"
server_key: "1b56b9befda35e7cf8884462b58eb9d2"
server_key_iv: "c4f05d5d3ad38a7ae24af6926d76422b"
server_key_version: "encryptionkey@0"
I20260927 10:57:20.615178 2066 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:57:20.617435 2080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:20.618592 2066 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 10:57:20.618691 2066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-1/wal
uuid: "5279f6b42e0a486baa534254db2bd8c7"
format_stamp: "Formatted at 2026-09-27 10:57:20 on dist-test-slave-tz51"
server_key: "1b56b9befda35e7cf8884462b58eb9d2"
server_key_iv: "c4f05d5d3ad38a7ae24af6926d76422b"
server_key_version: "encryptionkey@0"
I20260927 10:57:20.618777 2066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:20.643134 2066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:20.643887 2066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:20.644071 2066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:20.644692 2066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:20.645676 2066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:20.645721 2066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:20.645785 2066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:20.645823 2066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:20.654515 2066 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.2:40815
I20260927 10:57:20.654543 2193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.2:40815 every 8 connection(s)
I20260927 10:57:20.655412 2066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-1/data/info.pb
I20260927 10:57:20.661847 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 2066
I20260927 10:57:20.661984 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-1/wal/instance
I20260927 10:57:20.662294 14608 external_mini_cluster.cc:1428] Setting key 317c9394d7897456d2a26e489fa493f8
I20260927 10:57:20.664410 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.68.3:0
--local_ip_for_outbound_sockets=127.14.68.3
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46425
--builtin_ntp_servers=127.14.68.20:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:57:20.665181 1914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:20.657175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.2:59227 (local address 127.14.68.62:46425)
0927 10:57:20.657322 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:20.657327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:20.658499 (+ 1172us) server_negotiation.cc:407] Connection header received
0927 10:57:20.660048 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:20.660079 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:20.660136 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:20.660225 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:20.661071 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.661669 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.662998 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.663274 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.663376 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:20.664590 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:20.664596 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:20.664602 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:20.664618 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:20.664641 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:20.664653 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:20.664739 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:20.664878 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 10:57:20.664973 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:57:20.665851 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:20.657626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:46425 (local address 127.14.68.2:59227)
0927 10:57:20.658356 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:20.658396 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:20.659864 (+ 1468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:20.660403 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:20.660409 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:20.660491 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:20.660956 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:20.660967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.661792 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.661797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:20.662864 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:20.662874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.663409 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.663412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:20.663635 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:20.664311 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:57:20.664312 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:20.664350 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:57:20.664409 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:20.664739 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:20.664746 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:20.664750 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:57:20.664845 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 10:57:20.665056 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":105,"threads_started":1}
I20260927 10:57:20.666940 2194 heartbeater.cc:347] Connected to a master server at 127.14.68.62:46425
I20260927 10:57:20.667142 2194 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:20.667575 2194 heartbeater.cc:511] Master 127.14.68.62:46425 requested a full tablet report, sending...
I20260927 10:57:20.668536 1875 ts_manager.cc:194] Registered new tserver with Master: 5279f6b42e0a486baa534254db2bd8c7 (127.14.68.2:40815)
I20260927 10:57:20.669373 1875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.68.2:59227
W20260927 10:57:20.788872 2197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:20.789110 2197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:20.789146 2197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:20.793064 2197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:20.793112 2197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:20.793164 2197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.68.3
I20260927 10:57:20.797704 2197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.68.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.68.3
--webserver_port=0
--tserver_master_addrs=127.14.68.62:46425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.68.3
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:20.798612 2197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:20.799950 2197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:20.808907 2203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:20.808990 2202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:20.808951 2197 server_base.cc:1034] running on GCE node
W20260927 10:57:20.808898 2205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:20.809496 2197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:20.810322 2197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:20.811582 2197 hybrid_clock.cc:648] HybridClock initialized: now 1790506640811542 us; error 58 us; skew 500 ppm
I20260927 10:57:20.813503 2197 webserver.cc:460] Webserver started at http://127.14.68.3:40013/ using document root <none> and password file <none>
I20260927 10:57:20.814167 2197 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:20.814275 2197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:20.814497 2197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:20.817018 2197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-2/data/instance:
uuid: "7fd4a8b8ec3843d99c9ad94e47f84b01"
format_stamp: "Formatted at 2026-09-27 10:57:20 on dist-test-slave-tz51"
server_key: "17a7ac6cb0d5ec11163e36af8629d976"
server_key_iv: "d8b2f028dca040edb9e091dfa76e5782"
server_key_version: "encryptionkey@0"
I20260927 10:57:20.817675 2197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance:
uuid: "7fd4a8b8ec3843d99c9ad94e47f84b01"
format_stamp: "Formatted at 2026-09-27 10:57:20 on dist-test-slave-tz51"
server_key: "17a7ac6cb0d5ec11163e36af8629d976"
server_key_iv: "d8b2f028dca040edb9e091dfa76e5782"
server_key_version: "encryptionkey@0"
I20260927 10:57:20.822513 2197 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 10:57:20.825623 2211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:20.826823 2197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:57:20.826946 2197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-2/wal
uuid: "7fd4a8b8ec3843d99c9ad94e47f84b01"
format_stamp: "Formatted at 2026-09-27 10:57:20 on dist-test-slave-tz51"
server_key: "17a7ac6cb0d5ec11163e36af8629d976"
server_key_iv: "d8b2f028dca040edb9e091dfa76e5782"
server_key_version: "encryptionkey@0"
I20260927 10:57:20.827066 2197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:20.858758 2197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:20.859412 2197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:20.859638 2197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:20.860237 2197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:20.861135 2197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:20.861202 2197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:20.861268 2197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:20.861318 2197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:20.869668 2197 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.3:43911
I20260927 10:57:20.869742 2324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.3:43911 every 8 connection(s)
I20260927 10:57:20.870568 2197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-2/data/info.pb
I20260927 10:57:20.872277 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 2197
I20260927 10:57:20.872385 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-2/wal/instance
I20260927 10:57:20.872684 14608 external_mini_cluster.cc:1428] Setting key 3d8d86469affc63b3c141c85ac03f35c
I20260927 10:57:20.879766 1914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:20.872441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:33797 (local address 127.14.68.62:46425)
0927 10:57:20.872580 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:20.872583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:20.873878 (+ 1295us) server_negotiation.cc:407] Connection header received
0927 10:57:20.875500 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:20.875503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:20.875553 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:20.875621 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:20.876494 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.877009 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.878078 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.878283 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.878369 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:20.879328 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:20.879331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:20.879333 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:20.879344 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:20.879362 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:20.879370 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:20.879426 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:20.879565 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 10:57:20.879610 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:57:20.880571 2327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:20.872950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:46425 (local address 127.14.68.3:33797)
0927 10:57:20.873742 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:20.873782 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:20.875312 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:20.875815 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:20.875822 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:20.875904 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:20.876373 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:20.876385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.877114 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.877117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:20.877974 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:20.877982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.878415 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.878417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:20.878555 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:20.879103 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:57:20.879104 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:20.879141 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:57:20.879199 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:20.879441 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:20.879447 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:20.879451 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:57:20.879539 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 10:57:20.879817 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":145,"threads_started":1}
I20260927 10:57:20.881791 2325 heartbeater.cc:347] Connected to a master server at 127.14.68.62:46425
I20260927 10:57:20.882058 2325 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:20.882483 2325 heartbeater.cc:511] Master 127.14.68.62:46425 requested a full tablet report, sending...
I20260927 10:57:20.883380 1875 ts_manager.cc:194] Registered new tserver with Master: 7fd4a8b8ec3843d99c9ad94e47f84b01 (127.14.68.3:43911)
I20260927 10:57:20.883944 1875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.68.3:33797
I20260927 10:57:20.886363 14608 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:57:20.891786 1914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:20.888187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57410 (local address 127.14.68.62:46425)
0927 10:57:20.888297 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:20.888300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:20.888524 (+ 224us) server_negotiation.cc:407] Connection header received
0927 10:57:20.888664 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:20.888666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:20.888706 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:20.888794 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:20.889046 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.889530 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.890340 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.890574 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.890676 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:20.891349 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:20.891352 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:20.891355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:20.891366 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:20.891385 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:20.891398 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:20.891470 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:20.891589 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 10:57:20.891635 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:57:20.894161 1875 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:57:20.897074 1875 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:57:20.909914 2337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:20.904974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45626 (local address 127.14.68.3:43911)
0927 10:57:20.905304 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:20.905313 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:20.905333 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:57:20.905484 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:20.905489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:20.905691 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:20.905846 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:20.906337 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.907396 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.908712 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.909268 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.909398 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:20.909575 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:20.909689 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 10:57:20.909775 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":156,"threads_started":1}
I20260927 10:57:20.909907 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:20.904835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.3:43911 (local address 127.0.0.1:45626)
0927 10:57:20.905140 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:20.905182 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:20.905305 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:20.906023 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:20.906030 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:20.906071 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:20.906208 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:20.906215 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.907496 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.907503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:20.908578 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:20.908586 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.909398 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.909402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:20.909504 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:20.909548 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 10:57:20.909656 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 10:57:20.909768 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":97,"threads_started":1}
I20260927 10:57:20.911150 2338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:20.905546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:41261 (local address 127.0.0.1:42952)
0927 10:57:20.906137 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:20.906151 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:20.906229 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:20.906923 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:20.906929 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:20.906951 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:20.907034 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:20.907039 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.908497 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.908502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:20.909713 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:20.909723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.910767 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.910770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:20.910856 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:20.910878 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:57:20.910968 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 10:57:20.911046 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":82,"threads_started":1}
I20260927 10:57:20.911384 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:20.905833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:40815 (local address 127.0.0.1:35398)
0927 10:57:20.906340 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:20.906356 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:20.906437 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:20.907065 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:20.907068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:20.907085 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:20.907167 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:20.907172 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.908571 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.908574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:20.909784 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:20.909795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.910993 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.910997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:20.911095 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:20.911118 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:57:20.911194 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 10:57:20.911262 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":116,"thread_start_us":81,"threads_started":1}
I20260927 10:57:20.911780 2341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:20.906236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.14.68.2:40815)
0927 10:57:20.906618 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:20.906627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:20.906643 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:57:20.906698 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:20.906708 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:20.906839 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:20.906996 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:20.907287 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.908446 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.909902 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.910617 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.910726 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:20.910888 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:20.911428 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 10:57:20.911547 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":102,"threads_started":1}
I20260927 10:57:20.913108 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:20.905726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.14.68.1:41261)
0927 10:57:20.906483 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:20.906495 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:20.906518 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:57:20.906598 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:20.906603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:20.906774 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:20.906912 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:20.907155 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.908367 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.909849 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:20.910617 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:20.910735 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:20.910888 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:20.911005 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 10:57:20.912825 (+ 1820us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":113,"threads_started":1}
I20260927 10:57:20.913441 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 2e2abdfbc33a43c095243782b7428b0c (DEFAULT_TABLE table=test-table [id=db5c89827ca44eee879849646c0b4f2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:20.914465 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e2abdfbc33a43c095243782b7428b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:20.914387 2128 tablet_service.cc:1467] Processing CreateTablet for tablet 2e2abdfbc33a43c095243782b7428b0c (DEFAULT_TABLE table=test-table [id=db5c89827ca44eee879849646c0b4f2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:20.915401 2128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e2abdfbc33a43c095243782b7428b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:20.915652 1997 tablet_service.cc:1467] Processing CreateTablet for tablet 2e2abdfbc33a43c095243782b7428b0c (DEFAULT_TABLE table=test-table [id=db5c89827ca44eee879849646c0b4f2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:20.916610 1997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e2abdfbc33a43c095243782b7428b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:20.920811 2342 tablet_bootstrap.cc:492] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01: Bootstrap starting.
I20260927 10:57:20.922222 2343 tablet_bootstrap.cc:492] T 2e2abdfbc33a43c095243782b7428b0c P 5279f6b42e0a486baa534254db2bd8c7: Bootstrap starting.
I20260927 10:57:20.923190 2344 tablet_bootstrap.cc:492] T 2e2abdfbc33a43c095243782b7428b0c P ed55a26ff72f4a06a8a24a721f5de9a5: Bootstrap starting.
I20260927 10:57:20.923924 2342 tablet_bootstrap.cc:654] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:20.924521 2343 tablet_bootstrap.cc:654] T 2e2abdfbc33a43c095243782b7428b0c P 5279f6b42e0a486baa534254db2bd8c7: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:20.924796 2342 log.cc:824] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:20.925174 2344 tablet_bootstrap.cc:654] T 2e2abdfbc33a43c095243782b7428b0c P ed55a26ff72f4a06a8a24a721f5de9a5: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:20.925314 2343 log.cc:824] T 2e2abdfbc33a43c095243782b7428b0c P 5279f6b42e0a486baa534254db2bd8c7: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:20.925946 2344 log.cc:824] T 2e2abdfbc33a43c095243782b7428b0c P ed55a26ff72f4a06a8a24a721f5de9a5: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:20.926774 2342 tablet_bootstrap.cc:492] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01: No bootstrap required, opened a new log
I20260927 10:57:20.926966 2342 ts_tablet_manager.cc:1397] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:57:20.927266 2343 tablet_bootstrap.cc:492] T 2e2abdfbc33a43c095243782b7428b0c P 5279f6b42e0a486baa534254db2bd8c7: No bootstrap required, opened a new log
I20260927 10:57:20.927470 2343 ts_tablet_manager.cc:1397] T 2e2abdfbc33a43c095243782b7428b0c P 5279f6b42e0a486baa534254db2bd8c7: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:57:20.927942 2344 tablet_bootstrap.cc:492] T 2e2abdfbc33a43c095243782b7428b0c P ed55a26ff72f4a06a8a24a721f5de9a5: No bootstrap required, opened a new log
I20260927 10:57:20.928150 2344 ts_tablet_manager.cc:1397] T 2e2abdfbc33a43c095243782b7428b0c P ed55a26ff72f4a06a8a24a721f5de9a5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:57:20.930738 2343 raft_consensus.cc:348] T 2e2abdfbc33a43c095243782b7428b0c P 5279f6b42e0a486baa534254db2bd8c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd4a8b8ec3843d99c9ad94e47f84b01" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43911 } } peers { permanent_uuid: "ed55a26ff72f4a06a8a24a721f5de9a5" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41261 } } peers { permanent_uuid: "5279f6b42e0a486baa534254db2bd8c7" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 40815 } }
I20260927 10:57:20.930961 2343 raft_consensus.cc:374] T 2e2abdfbc33a43c095243782b7428b0c P 5279f6b42e0a486baa534254db2bd8c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:20.931048 2343 raft_consensus.cc:729] T 2e2abdfbc33a43c095243782b7428b0c P 5279f6b42e0a486baa534254db2bd8c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5279f6b42e0a486baa534254db2bd8c7, State: Initialized, Role: FOLLOWER
I20260927 10:57:20.931208 2342 raft_consensus.cc:348] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd4a8b8ec3843d99c9ad94e47f84b01" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43911 } } peers { permanent_uuid: "ed55a26ff72f4a06a8a24a721f5de9a5" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41261 } } peers { permanent_uuid: "5279f6b42e0a486baa534254db2bd8c7" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 40815 } }
I20260927 10:57:20.931442 2342 raft_consensus.cc:374] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:20.931519 2342 raft_consensus.cc:729] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd4a8b8ec3843d99c9ad94e47f84b01, State: Initialized, Role: FOLLOWER
I20260927 10:57:20.931533 2343 consensus_queue.cc:260] T 2e2abdfbc33a43c095243782b7428b0c P 5279f6b42e0a486baa534254db2bd8c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd4a8b8ec3843d99c9ad94e47f84b01" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43911 } } peers { permanent_uuid: "ed55a26ff72f4a06a8a24a721f5de9a5" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41261 } } peers { permanent_uuid: "5279f6b42e0a486baa534254db2bd8c7" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 40815 } }
I20260927 10:57:20.931958 2342 consensus_queue.cc:260] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd4a8b8ec3843d99c9ad94e47f84b01" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43911 } } peers { permanent_uuid: "ed55a26ff72f4a06a8a24a721f5de9a5" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41261 } } peers { permanent_uuid: "5279f6b42e0a486baa534254db2bd8c7" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 40815 } }
I20260927 10:57:20.932180 2344 raft_consensus.cc:348] T 2e2abdfbc33a43c095243782b7428b0c P ed55a26ff72f4a06a8a24a721f5de9a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd4a8b8ec3843d99c9ad94e47f84b01" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43911 } } peers { permanent_uuid: "ed55a26ff72f4a06a8a24a721f5de9a5" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41261 } } peers { permanent_uuid: "5279f6b42e0a486baa534254db2bd8c7" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 40815 } }
I20260927 10:57:20.932463 2344 raft_consensus.cc:374] T 2e2abdfbc33a43c095243782b7428b0c P ed55a26ff72f4a06a8a24a721f5de9a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:20.932554 2343 ts_tablet_manager.cc:1428] T 2e2abdfbc33a43c095243782b7428b0c P 5279f6b42e0a486baa534254db2bd8c7: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:57:20.932559 2344 raft_consensus.cc:729] T 2e2abdfbc33a43c095243782b7428b0c P ed55a26ff72f4a06a8a24a721f5de9a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed55a26ff72f4a06a8a24a721f5de9a5, State: Initialized, Role: FOLLOWER
I20260927 10:57:20.932618 2325 heartbeater.cc:503] Master 127.14.68.62:46425 was elected leader, sending a full tablet report...
I20260927 10:57:20.932880 2342 ts_tablet_manager.cc:1428] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:57:20.932910 2194 heartbeater.cc:503] Master 127.14.68.62:46425 was elected leader, sending a full tablet report...
I20260927 10:57:20.933898 2344 consensus_queue.cc:260] T 2e2abdfbc33a43c095243782b7428b0c P ed55a26ff72f4a06a8a24a721f5de9a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd4a8b8ec3843d99c9ad94e47f84b01" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43911 } } peers { permanent_uuid: "ed55a26ff72f4a06a8a24a721f5de9a5" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41261 } } peers { permanent_uuid: "5279f6b42e0a486baa534254db2bd8c7" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 40815 } }
I20260927 10:57:20.934490 2063 heartbeater.cc:503] Master 127.14.68.62:46425 was elected leader, sending a full tablet report...
I20260927 10:57:20.934768 2344 ts_tablet_manager.cc:1428] T 2e2abdfbc33a43c095243782b7428b0c P ed55a26ff72f4a06a8a24a721f5de9a5: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260927 10:57:20.964409 2064 tablet.cc:2369] T 2e2abdfbc33a43c095243782b7428b0c P ed55a26ff72f4a06a8a24a721f5de9a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:57:21.039585 2349 raft_consensus.cc:482] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:57:21.039861 2349 raft_consensus.cc:504] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd4a8b8ec3843d99c9ad94e47f84b01" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43911 } } peers { permanent_uuid: "ed55a26ff72f4a06a8a24a721f5de9a5" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41261 } } peers { permanent_uuid: "5279f6b42e0a486baa534254db2bd8c7" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 40815 } }
I20260927 10:57:21.040968 2349 leader_election.cc:288] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed55a26ff72f4a06a8a24a721f5de9a5 (127.14.68.1:41261), 5279f6b42e0a486baa534254db2bd8c7 (127.14.68.2:40815)
I20260927 10:57:21.044690 2327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:21.041210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:41261 (local address 127.14.68.3:37359)
0927 10:57:21.041350 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:21.041367 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:21.041471 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:21.041778 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:21.041781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:21.041798 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:21.041911 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:21.041917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:21.042626 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:21.042632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:21.043449 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:21.043458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:21.044151 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:21.044155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:21.044258 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:21.044279 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:57:21.044369 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 10:57:21.044562 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":18}
I20260927 10:57:21.044690 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:21.041340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:37359 (local address 127.14.68.1:41261)
0927 10:57:21.041494 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:21.041499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:21.041515 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:57:21.041576 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:21.041582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:21.041641 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:21.041746 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:21.042022 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:21.042518 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:21.043568 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:21.044033 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:21.044164 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:21.044225 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:21.044394 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 10:57:21.044562 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:57:21.045266 2017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e2abdfbc33a43c095243782b7428b0c" candidate_uuid: "7fd4a8b8ec3843d99c9ad94e47f84b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed55a26ff72f4a06a8a24a721f5de9a5" is_pre_election: true
I20260927 10:57:21.045464 2351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:21.041509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.2:40815 (local address 127.14.68.3:37105)
0927 10:57:21.041868 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:21.041879 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:21.041988 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:21.042229 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:21.042232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:21.042246 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:21.042374 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:21.042379 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:21.043067 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:21.043074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:21.044058 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:21.044066 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:21.045037 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:21.045041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:21.045133 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:21.045152 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 10:57:21.045261 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 10:57:21.045323 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":13,"thread_start_us":112,"threads_started":1}
I20260927 10:57:21.045455 2341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:21.041618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.68.3:37105 (local address 127.14.68.2:40815)
0927 10:57:21.041745 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:21.041751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:21.041961 (+ 210us) server_negotiation.cc:407] Connection header received
0927 10:57:21.042081 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:21.042083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:21.042131 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:21.042219 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:21.042482 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:21.042961 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:21.044231 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:21.044903 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:21.045028 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:21.045077 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:21.045289 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 10:57:21.045354 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:57:21.045593 2017 raft_consensus.cc:2454] T 2e2abdfbc33a43c095243782b7428b0c P ed55a26ff72f4a06a8a24a721f5de9a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fd4a8b8ec3843d99c9ad94e47f84b01 in term 0.
I20260927 10:57:21.046110 2212 leader_election.cc:302] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fd4a8b8ec3843d99c9ad94e47f84b01, ed55a26ff72f4a06a8a24a721f5de9a5; no voters:
I20260927 10:57:21.046373 2349 raft_consensus.cc:2781] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:57:21.046336 2148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e2abdfbc33a43c095243782b7428b0c" candidate_uuid: "7fd4a8b8ec3843d99c9ad94e47f84b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5279f6b42e0a486baa534254db2bd8c7" is_pre_election: true
I20260927 10:57:21.046466 2349 raft_consensus.cc:482] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:57:21.046527 2349 raft_consensus.cc:3037] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:21.046620 2148 raft_consensus.cc:2454] T 2e2abdfbc33a43c095243782b7428b0c P 5279f6b42e0a486baa534254db2bd8c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fd4a8b8ec3843d99c9ad94e47f84b01 in term 0.
I20260927 10:57:21.047622 2349 raft_consensus.cc:504] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd4a8b8ec3843d99c9ad94e47f84b01" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43911 } } peers { permanent_uuid: "ed55a26ff72f4a06a8a24a721f5de9a5" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41261 } } peers { permanent_uuid: "5279f6b42e0a486baa534254db2bd8c7" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 40815 } }
I20260927 10:57:21.047946 2349 leader_election.cc:288] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01 [CANDIDATE]: Term 1 election: Requested vote from peers ed55a26ff72f4a06a8a24a721f5de9a5 (127.14.68.1:41261), 5279f6b42e0a486baa534254db2bd8c7 (127.14.68.2:40815)
I20260927 10:57:21.048282 2017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e2abdfbc33a43c095243782b7428b0c" candidate_uuid: "7fd4a8b8ec3843d99c9ad94e47f84b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed55a26ff72f4a06a8a24a721f5de9a5"
I20260927 10:57:21.048292 2148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e2abdfbc33a43c095243782b7428b0c" candidate_uuid: "7fd4a8b8ec3843d99c9ad94e47f84b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5279f6b42e0a486baa534254db2bd8c7"
I20260927 10:57:21.048413 2148 raft_consensus.cc:3037] T 2e2abdfbc33a43c095243782b7428b0c P 5279f6b42e0a486baa534254db2bd8c7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:21.048403 2017 raft_consensus.cc:3037] T 2e2abdfbc33a43c095243782b7428b0c P ed55a26ff72f4a06a8a24a721f5de9a5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:21.049618 2148 raft_consensus.cc:2454] T 2e2abdfbc33a43c095243782b7428b0c P 5279f6b42e0a486baa534254db2bd8c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd4a8b8ec3843d99c9ad94e47f84b01 in term 1.
I20260927 10:57:21.049618 2017 raft_consensus.cc:2454] T 2e2abdfbc33a43c095243782b7428b0c P ed55a26ff72f4a06a8a24a721f5de9a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd4a8b8ec3843d99c9ad94e47f84b01 in term 1.
I20260927 10:57:21.049886 2212 leader_election.cc:302] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5279f6b42e0a486baa534254db2bd8c7, 7fd4a8b8ec3843d99c9ad94e47f84b01; no voters:
I20260927 10:57:21.050045 2349 raft_consensus.cc:2781] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:21.050269 2349 raft_consensus.cc:686] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01 [term 1 LEADER]: Becoming Leader. State: Replica: 7fd4a8b8ec3843d99c9ad94e47f84b01, State: Running, Role: LEADER
I20260927 10:57:21.050587 2349 consensus_queue.cc:237] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd4a8b8ec3843d99c9ad94e47f84b01" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43911 } } peers { permanent_uuid: "ed55a26ff72f4a06a8a24a721f5de9a5" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41261 } } peers { permanent_uuid: "5279f6b42e0a486baa534254db2bd8c7" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 40815 } }
I20260927 10:57:21.053622 1873 catalog_manager.cc:5495] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fd4a8b8ec3843d99c9ad94e47f84b01 (127.14.68.3). New cstate: current_term: 1 leader_uuid: "7fd4a8b8ec3843d99c9ad94e47f84b01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd4a8b8ec3843d99c9ad94e47f84b01" member_type: VOTER last_known_addr { host: "127.14.68.3" port: 43911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed55a26ff72f4a06a8a24a721f5de9a5" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5279f6b42e0a486baa534254db2bd8c7" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 40815 } health_report { overall_health: UNKNOWN } } }
I20260927 10:57:21.098923 2337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:21.095485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45630 (local address 127.14.68.3:43911)
0927 10:57:21.095621 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:21.095625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:21.095640 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:57:21.095781 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:21.095784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:21.095835 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:21.095945 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:57:21.096174 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:21.096673 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:21.097529 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:21.097751 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:21.097861 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:21.098152 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:57:21.098535 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:57:21.098690 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:21.098724 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 10:57:21.098769 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:57:21.104395 2148 raft_consensus.cc:1261] T 2e2abdfbc33a43c095243782b7428b0c P 5279f6b42e0a486baa534254db2bd8c7 [term 1 FOLLOWER]: Refusing update from remote peer 7fd4a8b8ec3843d99c9ad94e47f84b01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:21.104396 2017 raft_consensus.cc:1261] T 2e2abdfbc33a43c095243782b7428b0c P ed55a26ff72f4a06a8a24a721f5de9a5 [term 1 FOLLOWER]: Refusing update from remote peer 7fd4a8b8ec3843d99c9ad94e47f84b01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:21.105095 2352 consensus_queue.cc:1035] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5279f6b42e0a486baa534254db2bd8c7" member_type: VOTER last_known_addr { host: "127.14.68.2" port: 40815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:21.105248 2349 consensus_queue.cc:1035] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed55a26ff72f4a06a8a24a721f5de9a5" member_type: VOTER last_known_addr { host: "127.14.68.1" port: 41261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:21.112680 2356 mvcc.cc:205] Tried to move back new op lower bound from 7333915201956425728 to 7333915201745625088. Current Snapshot: MvccSnapshot[applied={T|T < 7333915201956425728}]
I20260927 10:57:21.120203 1873 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:57:21.120373 1873 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:57:21.122748 1873 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6cc2d1723d324a74a8516c9d3c239a03: Sending DeleteTablet for 3 replicas of tablet 2e2abdfbc33a43c095243782b7428b0c
I20260927 10:57:21.123518 1997 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e2abdfbc33a43c095243782b7428b0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:21 UTC) from {username='slave'} at 127.0.0.1:42952
I20260927 10:57:21.123540 2128 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e2abdfbc33a43c095243782b7428b0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:21 UTC) from {username='slave'} at 127.0.0.1:35398
I20260927 10:57:21.123540 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e2abdfbc33a43c095243782b7428b0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:21 UTC) from {username='slave'} at 127.0.0.1:45626
I20260927 10:57:21.123998 2365 tablet_replica.cc:330] T 2e2abdfbc33a43c095243782b7428b0c P 5279f6b42e0a486baa534254db2bd8c7: stopping tablet replica
I20260927 10:57:21.124274 2367 tablet_replica.cc:330] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01: stopping tablet replica
I20260927 10:57:21.124332 2365 raft_consensus.cc:2229] T 2e2abdfbc33a43c095243782b7428b0c P 5279f6b42e0a486baa534254db2bd8c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:57:21.124521 2367 raft_consensus.cc:2229] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:57:21.124598 2365 raft_consensus.cc:2258] T 2e2abdfbc33a43c095243782b7428b0c P 5279f6b42e0a486baa534254db2bd8c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:21.124905 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 1935
I20260927 10:57:21.125438 2367 raft_consensus.cc:2258] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:21.126255 2365 ts_tablet_manager.cc:1905] T 2e2abdfbc33a43c095243782b7428b0c P 5279f6b42e0a486baa534254db2bd8c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:57:21.126683 2367 ts_tablet_manager.cc:1905] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:57:21.128980 2365 ts_tablet_manager.cc:1918] T 2e2abdfbc33a43c095243782b7428b0c P 5279f6b42e0a486baa534254db2bd8c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:57:21.129096 2365 log.cc:1196] T 2e2abdfbc33a43c095243782b7428b0c P 5279f6b42e0a486baa534254db2bd8c7: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-1/wal/wals/2e2abdfbc33a43c095243782b7428b0c
I20260927 10:57:21.129432 2365 ts_tablet_manager.cc:1939] T 2e2abdfbc33a43c095243782b7428b0c P 5279f6b42e0a486baa534254db2bd8c7: Deleting consensus metadata
I20260927 10:57:21.129658 2367 ts_tablet_manager.cc:1918] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:57:21.129796 2367 log.cc:1196] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01: Deleting WAL directory at /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506389460539-14608-0/minicluster-data/ts-2/wal/wals/2e2abdfbc33a43c095243782b7428b0c
I20260927 10:57:21.130101 2367 ts_tablet_manager.cc:1939] T 2e2abdfbc33a43c095243782b7428b0c P 7fd4a8b8ec3843d99c9ad94e47f84b01: Deleting consensus metadata
I20260927 10:57:21.130646 1859 catalog_manager.cc:4841] TS 5279f6b42e0a486baa534254db2bd8c7 (127.14.68.2:40815): tablet 2e2abdfbc33a43c095243782b7428b0c (table test-table [id=db5c89827ca44eee879849646c0b4f2e]) successfully deleted
I20260927 10:57:21.131250 1859 catalog_manager.cc:4841] TS 7fd4a8b8ec3843d99c9ad94e47f84b01 (127.14.68.3:43911): tablet 2e2abdfbc33a43c095243782b7428b0c (table test-table [id=db5c89827ca44eee879849646c0b4f2e]) successfully deleted
W20260927 10:57:21.134109 1859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.68.1:41261 (error 108)
I20260927 10:57:21.134860 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 2066
I20260927 10:57:21.135053 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:21.134768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.1:41261 (local address 127.0.0.1:42968)
0927 10:57:21.134889 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:21.134961 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.1:41261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 10:57:21.135587 1859 catalog_manager.cc:4584] TS ed55a26ff72f4a06a8a24a721f5de9a5 (127.14.68.1:41261): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2e2abdfbc33a43c095243782b7428b0c: Network error: Client connection negotiation failed: client connection to 127.14.68.1:41261: connect: Connection refused (error 111)
I20260927 10:57:21.140955 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 2197
I20260927 10:57:21.145875 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 1844
2026-09-27T10:57:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1105 ms)
[----------] 6 tests from EncryptionPolicyTest (10582 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:21 dist-test-slave-tz51 krb5kdc[2371](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:21 dist-test-slave-tz51 krb5kdc[2371](info): set up 2 sockets
Sep 27 10:57:21 dist-test-slave-tz51 krb5kdc[2371](info): commencing operation
krb5kdc: starting...
W20260927 10:57:22.175863 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:57:22 dist-test-slave-tz51 krb5kdc[2371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:57:22.294200 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46645
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:22.402477 2384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:22.402732 2384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:22.402773 2384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:22.406639 2384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:22.406688 2384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:22.406708 2384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:22.406724 2384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:22.406738 2384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:22.411720 2384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46645
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:22.412643 2384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:22.413590 2384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:22.421799 2390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:22.421806 2392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:22.421806 2389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:22.421890 2384 server_base.cc:1034] running on GCE node
I20260927 10:57:22.422703 2384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:22.423784 2384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:22.424963 2384 hybrid_clock.cc:648] HybridClock initialized: now 1790506642424954 us; error 28 us; skew 500 ppm
Sep 27 10:57:22 dist-test-slave-tz51 krb5kdc[2371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:22.427527 2384 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:57:22.428345 2384 webserver.cc:460] Webserver started at http://127.0.0.1:37975/ using document root <none> and password file <none>
I20260927 10:57:22.428819 2384 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:22.428874 2384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:22.429042 2384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:22.430884 2384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "6768b343935c48cc92a3bd3f89ee0bd1"
format_stamp: "Formatted at 2026-09-27 10:57:22 on dist-test-slave-tz51"
server_key: "6ad5ae6548292d9e855e192e81a15c14"
server_key_iv: "8487f20c2cbbe3f3c7546685f79ef225"
server_key_version: "encryptionkey@0"
I20260927 10:57:22.431376 2384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "6768b343935c48cc92a3bd3f89ee0bd1"
format_stamp: "Formatted at 2026-09-27 10:57:22 on dist-test-slave-tz51"
server_key: "6ad5ae6548292d9e855e192e81a15c14"
server_key_iv: "8487f20c2cbbe3f3c7546685f79ef225"
server_key_version: "encryptionkey@0"
I20260927 10:57:22.434834 2384 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 10:57:22.437129 2399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:22.438285 2384 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:57:22.438405 2384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "6768b343935c48cc92a3bd3f89ee0bd1"
format_stamp: "Formatted at 2026-09-27 10:57:22 on dist-test-slave-tz51"
server_key: "6ad5ae6548292d9e855e192e81a15c14"
server_key_iv: "8487f20c2cbbe3f3c7546685f79ef225"
server_key_version: "encryptionkey@0"
I20260927 10:57:22.438504 2384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:22.463025 2384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:22.465782 2384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:22.465976 2384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:22.472679 2384 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46645
I20260927 10:57:22.472712 2451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46645 every 8 connection(s)
I20260927 10:57:22.473562 2384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:22.476167 2452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:22.479955 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 2384
I20260927 10:57:22.480088 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:22.480399 14608 external_mini_cluster.cc:1428] Setting key 40ff844f620307b4af743304ab8b763e
I20260927 10:57:22.481789 2452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6768b343935c48cc92a3bd3f89ee0bd1: Bootstrap starting.
I20260927 10:57:22.483978 2452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6768b343935c48cc92a3bd3f89ee0bd1: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:22.484681 2452 log.cc:824] T 00000000000000000000000000000000 P 6768b343935c48cc92a3bd3f89ee0bd1: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:22.486420 2452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6768b343935c48cc92a3bd3f89ee0bd1: No bootstrap required, opened a new log
Sep 27 10:57:22 dist-test-slave-tz51 krb5kdc[2371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 10:57:22.489825 2452 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6768b343935c48cc92a3bd3f89ee0bd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6768b343935c48cc92a3bd3f89ee0bd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46645 } }
I20260927 10:57:22.490044 2452 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6768b343935c48cc92a3bd3f89ee0bd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:22.490104 2452 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6768b343935c48cc92a3bd3f89ee0bd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6768b343935c48cc92a3bd3f89ee0bd1, State: Initialized, Role: FOLLOWER
I20260927 10:57:22.490564 2452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6768b343935c48cc92a3bd3f89ee0bd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6768b343935c48cc92a3bd3f89ee0bd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46645 } }
I20260927 10:57:22.490718 2452 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6768b343935c48cc92a3bd3f89ee0bd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:22.490799 2452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6768b343935c48cc92a3bd3f89ee0bd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:22.490895 2452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6768b343935c48cc92a3bd3f89ee0bd1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:22.491670 2452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6768b343935c48cc92a3bd3f89ee0bd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6768b343935c48cc92a3bd3f89ee0bd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46645 } }
I20260927 10:57:22.492031 2452 leader_election.cc:302] T 00000000000000000000000000000000 P 6768b343935c48cc92a3bd3f89ee0bd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6768b343935c48cc92a3bd3f89ee0bd1; no voters:
I20260927 10:57:22.492341 2452 leader_election.cc:288] T 00000000000000000000000000000000 P 6768b343935c48cc92a3bd3f89ee0bd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:22.492399 2457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6768b343935c48cc92a3bd3f89ee0bd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:22.492586 2457 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6768b343935c48cc92a3bd3f89ee0bd1 [term 1 LEADER]: Becoming Leader. State: Replica: 6768b343935c48cc92a3bd3f89ee0bd1, State: Running, Role: LEADER
I20260927 10:57:22.492865 2457 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6768b343935c48cc92a3bd3f89ee0bd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6768b343935c48cc92a3bd3f89ee0bd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46645 } }
I20260927 10:57:22.493522 2452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6768b343935c48cc92a3bd3f89ee0bd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:22.493947 2458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6768b343935c48cc92a3bd3f89ee0bd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6768b343935c48cc92a3bd3f89ee0bd1. Latest consensus state: current_term: 1 leader_uuid: "6768b343935c48cc92a3bd3f89ee0bd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6768b343935c48cc92a3bd3f89ee0bd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46645 } } }
I20260927 10:57:22.493979 2457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6768b343935c48cc92a3bd3f89ee0bd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6768b343935c48cc92a3bd3f89ee0bd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6768b343935c48cc92a3bd3f89ee0bd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46645 } } }
I20260927 10:57:22.494061 2458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6768b343935c48cc92a3bd3f89ee0bd1 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:22.494768 2455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:22.481756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56367 (local address 127.0.0.1:46645)
0927 10:57:22.482178 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:22.482193 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:22.482227 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:57:22.482839 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:22.482853 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:22.483173 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:22.483490 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:22.484389 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:22.485421 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:22.486301 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:22.486816 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:22.486975 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:22.489526 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:22.489531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:22.489543 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:22.489567 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:22.491493 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:22.492077 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:22.492083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:22.492088 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:22.492243 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:22.492604 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:22.492607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:22.492609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:22.493055 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:22.493280 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:22.493589 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 10:57:22.493798 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":133,"threads_started":1}
I20260927 10:57:22.494829 2461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:22.494068 2457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6768b343935c48cc92a3bd3f89ee0bd1 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:22.497824 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:57:22.498620 2461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:22.504267 2461 catalog_manager.cc:1347] Generated new cluster ID: 9742948d91f84c5ea9920bdc467cddb8
I20260927 10:57:22.504369 2461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:22.508308 2455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:22.500110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42177 (local address 127.0.0.1:46645)
0927 10:57:22.500201 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:22.500205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:22.500558 (+ 353us) server_negotiation.cc:407] Connection header received
0927 10:57:22.500736 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:22.500739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:22.500777 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:22.500865 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:22.501467 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:22.501964 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:22.502972 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:22.503321 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:22.503391 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:22.505023 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:22.505027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:22.505029 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:22.505052 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:22.506503 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:22.506966 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:22.506970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:22.506971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:22.507046 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:22.507367 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:22.507370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:22.507372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:22.507531 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:22.507679 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:22.507975 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 10:57:22.508063 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 10:57:22.522791 2461 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:22.523814 2461 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:22.534116 2461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6768b343935c48cc92a3bd3f89ee0bd1: Generated new TSK 0
I20260927 10:57:22.534757 2461 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:57:22.540297 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 2384
W20260927 10:57:22.545696 2369 connection.cc:535] client connection to 127.0.0.1:46645 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46645): Connection reset by peer (error 104)
2026-09-27T10:57:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:22 dist-test-slave-tz51 krb5kdc[2483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:22 dist-test-slave-tz51 krb5kdc[2483](info): set up 2 sockets
Sep 27 10:57:22 dist-test-slave-tz51 krb5kdc[2483](info): commencing operation
krb5kdc: starting...
W20260927 10:57:23.575776 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:57:23 dist-test-slave-tz51 krb5kdc[2483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:57:23.694794 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45139
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:23.801609 2496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:23.801914 2496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:23.801992 2496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:23.805960 2496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:23.806041 2496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:23.806068 2496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:23.806089 2496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:23.806109 2496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:23.811396 2496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45139
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:23.812507 2496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:23.813490 2496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:23.821933 2504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:23.821954 2502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:23.821938 2501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:23.821933 2496 server_base.cc:1034] running on GCE node
I20260927 10:57:23.822629 2496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:23.823763 2496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:23.824961 2496 hybrid_clock.cc:648] HybridClock initialized: now 1790506643824940 us; error 36 us; skew 500 ppm
Sep 27 10:57:23 dist-test-slave-tz51 krb5kdc[2483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:23.827742 2496 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:57:23.828635 2496 webserver.cc:460] Webserver started at http://127.0.0.1:40351/ using document root <none> and password file <none>
I20260927 10:57:23.829181 2496 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:23.829269 2496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:23.829499 2496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:23.831468 2496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "9df39c60cf3f42c184a6f0e3ca41dfbf"
format_stamp: "Formatted at 2026-09-27 10:57:23 on dist-test-slave-tz51"
server_key: "206f7bcb34680743e6efd295bd041900"
server_key_iv: "4894cbc51ebc042952c4a27e41fa1b16"
server_key_version: "encryptionkey@0"
I20260927 10:57:23.832070 2496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "9df39c60cf3f42c184a6f0e3ca41dfbf"
format_stamp: "Formatted at 2026-09-27 10:57:23 on dist-test-slave-tz51"
server_key: "206f7bcb34680743e6efd295bd041900"
server_key_iv: "4894cbc51ebc042952c4a27e41fa1b16"
server_key_version: "encryptionkey@0"
I20260927 10:57:23.835577 2496 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:57:23.837817 2511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:23.838989 2496 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:23.839121 2496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "9df39c60cf3f42c184a6f0e3ca41dfbf"
format_stamp: "Formatted at 2026-09-27 10:57:23 on dist-test-slave-tz51"
server_key: "206f7bcb34680743e6efd295bd041900"
server_key_iv: "4894cbc51ebc042952c4a27e41fa1b16"
server_key_version: "encryptionkey@0"
I20260927 10:57:23.839238 2496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:23.848515 2496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:23.851586 2496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:23.851810 2496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:23.858299 2496 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45139
I20260927 10:57:23.858325 2563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45139 every 8 connection(s)
I20260927 10:57:23.859155 2496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:23.860190 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 2496
I20260927 10:57:23.860324 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:23.860627 14608 external_mini_cluster.cc:1428] Setting key 0a4551e11e422d69ccc5f8bf972e332a
I20260927 10:57:23.862087 2564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:23.867244 2564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df39c60cf3f42c184a6f0e3ca41dfbf: Bootstrap starting.
I20260927 10:57:23.868999 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:23.861904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44759 (local address 127.0.0.1:45139)
0927 10:57:23.862349 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:23.862363 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:23.862396 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:57:23.862991 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:23.863005 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:23.863321 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:23.863631 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:23.864045 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:23.864977 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:23.865861 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:23.866337 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:23.866498 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:23.867389 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:23.867395 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:23.867410 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:23.867841 (+ 431us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:23.867893 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:23.867915 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:23.868050 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:23.868239 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 10:57:23.868362 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":144,"threads_started":1}
I20260927 10:57:23.869475 2564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9df39c60cf3f42c184a6f0e3ca41dfbf: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:23.870150 2564 log.cc:824] T 00000000000000000000000000000000 P 9df39c60cf3f42c184a6f0e3ca41dfbf: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:23.871745 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:57:23.872099 2564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df39c60cf3f42c184a6f0e3ca41dfbf: No bootstrap required, opened a new log
I20260927 10:57:23.876161 2564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9df39c60cf3f42c184a6f0e3ca41dfbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df39c60cf3f42c184a6f0e3ca41dfbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } }
I20260927 10:57:23.876396 2564 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9df39c60cf3f42c184a6f0e3ca41dfbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:23.876479 2564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9df39c60cf3f42c184a6f0e3ca41dfbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9df39c60cf3f42c184a6f0e3ca41dfbf, State: Initialized, Role: FOLLOWER
I20260927 10:57:23.876948 2564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9df39c60cf3f42c184a6f0e3ca41dfbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df39c60cf3f42c184a6f0e3ca41dfbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } }
I20260927 10:57:23.877121 2564 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9df39c60cf3f42c184a6f0e3ca41dfbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:23.877220 2564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9df39c60cf3f42c184a6f0e3ca41dfbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:23.877331 2564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9df39c60cf3f42c184a6f0e3ca41dfbf [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:23.878011 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:23.873646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56423 (local address 127.0.0.1:45139)
0927 10:57:23.873753 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:23.873756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:23.873989 (+ 233us) server_negotiation.cc:407] Connection header received
0927 10:57:23.874112 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:23.874114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:23.874144 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:23.874208 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:23.874509 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:23.874964 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:23.876054 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:23.876554 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:23.876659 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:23.877575 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:23.877578 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:23.877580 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:23.877590 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:23.877611 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:23.877620 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:23.877689 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:23.877834 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 10:57:23.877859 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 10:57:23.878420 2564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9df39c60cf3f42c184a6f0e3ca41dfbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df39c60cf3f42c184a6f0e3ca41dfbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } }
I20260927 10:57:23.878744 2564 leader_election.cc:302] T 00000000000000000000000000000000 P 9df39c60cf3f42c184a6f0e3ca41dfbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9df39c60cf3f42c184a6f0e3ca41dfbf; no voters:
I20260927 10:57:23.879053 2564 leader_election.cc:288] T 00000000000000000000000000000000 P 9df39c60cf3f42c184a6f0e3ca41dfbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:23.879148 2574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9df39c60cf3f42c184a6f0e3ca41dfbf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:23.879364 2574 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9df39c60cf3f42c184a6f0e3ca41dfbf [term 1 LEADER]: Becoming Leader. State: Replica: 9df39c60cf3f42c184a6f0e3ca41dfbf, State: Running, Role: LEADER
I20260927 10:57:23.879731 2574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9df39c60cf3f42c184a6f0e3ca41dfbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df39c60cf3f42c184a6f0e3ca41dfbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } }
I20260927 10:57:23.880244 2564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9df39c60cf3f42c184a6f0e3ca41dfbf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:23.881362 2575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df39c60cf3f42c184a6f0e3ca41dfbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9df39c60cf3f42c184a6f0e3ca41dfbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df39c60cf3f42c184a6f0e3ca41dfbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } } }
I20260927 10:57:23.881362 2576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df39c60cf3f42c184a6f0e3ca41dfbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9df39c60cf3f42c184a6f0e3ca41dfbf. Latest consensus state: current_term: 1 leader_uuid: "9df39c60cf3f42c184a6f0e3ca41dfbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df39c60cf3f42c184a6f0e3ca41dfbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } } }
I20260927 10:57:23.881502 2575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df39c60cf3f42c184a6f0e3ca41dfbf [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:23.881502 2576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df39c60cf3f42c184a6f0e3ca41dfbf [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:23.882550 2584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:23.885350 2584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:23.890322 2584 catalog_manager.cc:1347] Generated new cluster ID: 8b2c75459f7c439892bd52c99799b405
I20260927 10:57:23.890424 2584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:23.906961 2584 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:23.907872 2584 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:23.914474 2584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9df39c60cf3f42c184a6f0e3ca41dfbf: Generated new TSK 0
I20260927 10:57:23.915089 2584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:57:23.921157 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 2496
W20260927 10:57:23.926481 2481 connection.cc:535] client connection to 127.0.0.1:45139 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45139): Connection reset by peer (error 104)
2026-09-27T10:57:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:23 dist-test-slave-tz51 krb5kdc[2595](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:23 dist-test-slave-tz51 krb5kdc[2595](info): set up 2 sockets
Sep 27 10:57:23 dist-test-slave-tz51 krb5kdc[2595](info): commencing operation
krb5kdc: starting...
W20260927 10:57:24.948097 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:57:25 dist-test-slave-tz51 krb5kdc[2595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:57:25.066211 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35089
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:25.173346 2608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:25.173702 2608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:25.173784 2608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:25.177680 2608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:25.177759 2608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:25.177798 2608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:25.177855 2608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:25.177896 2608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:25.182829 2608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35089
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:25.183794 2608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:25.184719 2608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:25.192672 2614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:25.192672 2616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:25.192868 2608 server_base.cc:1034] running on GCE node
W20260927 10:57:25.192695 2613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:25.193414 2608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:25.194412 2608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:25.195597 2608 hybrid_clock.cc:648] HybridClock initialized: now 1790506645195580 us; error 28 us; skew 500 ppm
Sep 27 10:57:25 dist-test-slave-tz51 krb5kdc[2595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:25.198114 2608 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:57:25.198913 2608 webserver.cc:460] Webserver started at http://127.0.0.1:38877/ using document root <none> and password file <none>
I20260927 10:57:25.199422 2608 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:25.199508 2608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:25.199782 2608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:25.201671 2608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "b99172c5dbeb441fa14673e959326fb0"
format_stamp: "Formatted at 2026-09-27 10:57:25 on dist-test-slave-tz51"
server_key: "c92a0ddcd36c6b2400777a9894a86704"
server_key_iv: "44078f39fd960b7c79dc34fcd16b92f4"
server_key_version: "encryptionkey@0"
I20260927 10:57:25.202206 2608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "b99172c5dbeb441fa14673e959326fb0"
format_stamp: "Formatted at 2026-09-27 10:57:25 on dist-test-slave-tz51"
server_key: "c92a0ddcd36c6b2400777a9894a86704"
server_key_iv: "44078f39fd960b7c79dc34fcd16b92f4"
server_key_version: "encryptionkey@0"
I20260927 10:57:25.205651 2608 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 10:57:25.207839 2623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:25.208918 2608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:25.209048 2608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "b99172c5dbeb441fa14673e959326fb0"
format_stamp: "Formatted at 2026-09-27 10:57:25 on dist-test-slave-tz51"
server_key: "c92a0ddcd36c6b2400777a9894a86704"
server_key_iv: "44078f39fd960b7c79dc34fcd16b92f4"
server_key_version: "encryptionkey@0"
I20260927 10:57:25.209146 2608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:25.213409 2608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:25.215941 2608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:25.216117 2608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:25.222698 2608 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35089
I20260927 10:57:25.222719 2675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35089 every 8 connection(s)
I20260927 10:57:25.223495 2608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:25.225977 2676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:25.230911 2676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b99172c5dbeb441fa14673e959326fb0: Bootstrap starting.
I20260927 10:57:25.231420 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 2608
I20260927 10:57:25.231551 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:25.231885 14608 external_mini_cluster.cc:1428] Setting key e30027f6f946410e2a5d50b2be824d2e
I20260927 10:57:25.233625 2676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b99172c5dbeb441fa14673e959326fb0: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:25.234301 2676 log.cc:824] T 00000000000000000000000000000000 P b99172c5dbeb441fa14673e959326fb0: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:25.236433 2676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b99172c5dbeb441fa14673e959326fb0: No bootstrap required, opened a new log
I20260927 10:57:25.236869 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:25.233121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36661 (local address 127.0.0.1:35089)
0927 10:57:25.233531 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:25.233545 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:25.233577 (+ 32us) server_negotiation.cc:407] Connection header received
0927 10:57:25.234214 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:25.234230 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:25.234534 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:25.234891 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:25.235655 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:25.235661 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:25.235671 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:25.235679 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:25.235776 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:25.235797 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:25.235880 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:25.236018 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 10:57:25.236193 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":118,"threads_started":1}
I20260927 10:57:25.239602 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:57:25.240509 2676 raft_consensus.cc:348] T 00000000000000000000000000000000 P b99172c5dbeb441fa14673e959326fb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b99172c5dbeb441fa14673e959326fb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35089 } }
I20260927 10:57:25.240760 2676 raft_consensus.cc:374] T 00000000000000000000000000000000 P b99172c5dbeb441fa14673e959326fb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:25.240828 2676 raft_consensus.cc:729] T 00000000000000000000000000000000 P b99172c5dbeb441fa14673e959326fb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b99172c5dbeb441fa14673e959326fb0, State: Initialized, Role: FOLLOWER
I20260927 10:57:25.241305 2676 consensus_queue.cc:260] T 00000000000000000000000000000000 P b99172c5dbeb441fa14673e959326fb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b99172c5dbeb441fa14673e959326fb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35089 } }
I20260927 10:57:25.241456 2676 raft_consensus.cc:388] T 00000000000000000000000000000000 P b99172c5dbeb441fa14673e959326fb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:25.241528 2676 raft_consensus.cc:482] T 00000000000000000000000000000000 P b99172c5dbeb441fa14673e959326fb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:25.241632 2676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b99172c5dbeb441fa14673e959326fb0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:25.242547 2676 raft_consensus.cc:504] T 00000000000000000000000000000000 P b99172c5dbeb441fa14673e959326fb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b99172c5dbeb441fa14673e959326fb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35089 } }
I20260927 10:57:25.242977 2676 leader_election.cc:302] T 00000000000000000000000000000000 P b99172c5dbeb441fa14673e959326fb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b99172c5dbeb441fa14673e959326fb0; no voters:
I20260927 10:57:25.243311 2676 leader_election.cc:288] T 00000000000000000000000000000000 P b99172c5dbeb441fa14673e959326fb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:25.243435 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:25.241755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59283 (local address 127.0.0.1:35089)
0927 10:57:25.241840 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:25.241843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:25.242026 (+ 183us) server_negotiation.cc:407] Connection header received
0927 10:57:25.242141 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:25.242144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:25.242191 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:25.242278 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:25.242973 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:25.242975 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:25.242977 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:25.242987 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:25.243008 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:25.243017 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:25.243082 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:25.243213 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 10:57:25.243251 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 10:57:25.243516 2686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b99172c5dbeb441fa14673e959326fb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:25.243768 2686 raft_consensus.cc:686] T 00000000000000000000000000000000 P b99172c5dbeb441fa14673e959326fb0 [term 1 LEADER]: Becoming Leader. State: Replica: b99172c5dbeb441fa14673e959326fb0, State: Running, Role: LEADER
I20260927 10:57:25.244067 2686 consensus_queue.cc:237] T 00000000000000000000000000000000 P b99172c5dbeb441fa14673e959326fb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b99172c5dbeb441fa14673e959326fb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35089 } }
I20260927 10:57:25.244488 2676 sys_catalog.cc:564] T 00000000000000000000000000000000 P b99172c5dbeb441fa14673e959326fb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:25.245802 2687 sys_catalog.cc:455] T 00000000000000000000000000000000 P b99172c5dbeb441fa14673e959326fb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b99172c5dbeb441fa14673e959326fb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b99172c5dbeb441fa14673e959326fb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35089 } } }
I20260927 10:57:25.245811 2688 sys_catalog.cc:455] T 00000000000000000000000000000000 P b99172c5dbeb441fa14673e959326fb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b99172c5dbeb441fa14673e959326fb0. Latest consensus state: current_term: 1 leader_uuid: "b99172c5dbeb441fa14673e959326fb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b99172c5dbeb441fa14673e959326fb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35089 } } }
I20260927 10:57:25.245937 2687 sys_catalog.cc:458] T 00000000000000000000000000000000 P b99172c5dbeb441fa14673e959326fb0 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:25.245946 2688 sys_catalog.cc:458] T 00000000000000000000000000000000 P b99172c5dbeb441fa14673e959326fb0 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:25.246315 2697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:25.248970 2697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:25.253692 2697 catalog_manager.cc:1347] Generated new cluster ID: 2eeb6781461349e7bf3a397fbc294a56
I20260927 10:57:25.253770 2697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:25.274664 2697 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:25.274998 2697 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:25.283265 2697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b99172c5dbeb441fa14673e959326fb0: Generated new TSK 0
I20260927 10:57:25.283908 2697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:57:25.290535 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 2608
2026-09-27T10:57:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:25 dist-test-slave-tz51 krb5kdc[2707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:25 dist-test-slave-tz51 krb5kdc[2707](info): set up 2 sockets
Sep 27 10:57:25 dist-test-slave-tz51 krb5kdc[2707](info): commencing operation
krb5kdc: starting...
W20260927 10:57:26.323774 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:57:26 dist-test-slave-tz51 krb5kdc[2707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:57:26.440892 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:26.546495 2720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:26.546728 2720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:26.546769 2720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:26.550582 2720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:26.550630 2720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:26.550649 2720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:26.550664 2720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:26.550678 2720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:26.555631 2720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:26.556553 2720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:26.557533 2720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:26.565469 2728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:26.565423 2726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:26.565424 2725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:26.565838 2720 server_base.cc:1034] running on GCE node
I20260927 10:57:26.566336 2720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:26.567313 2720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:26.568461 2720 hybrid_clock.cc:648] HybridClock initialized: now 1790506646568435 us; error 38 us; skew 500 ppm
Sep 27 10:57:26 dist-test-slave-tz51 krb5kdc[2707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:26.574281 2720 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:57:26.575135 2720 webserver.cc:460] Webserver started at http://127.0.0.1:35765/ using document root <none> and password file <none>
I20260927 10:57:26.575664 2720 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:26.575801 2720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:26.576025 2720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:26.577965 2720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "32d38f08af674940a7227b997a238338"
format_stamp: "Formatted at 2026-09-27 10:57:26 on dist-test-slave-tz51"
server_key: "430d0c9ebde553c6e4a7806ca5c3c784"
server_key_iv: "3f4b0bae1e15177a743632690496118e"
server_key_version: "encryptionkey@0"
I20260927 10:57:26.578508 2720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "32d38f08af674940a7227b997a238338"
format_stamp: "Formatted at 2026-09-27 10:57:26 on dist-test-slave-tz51"
server_key: "430d0c9ebde553c6e4a7806ca5c3c784"
server_key_iv: "3f4b0bae1e15177a743632690496118e"
server_key_version: "encryptionkey@0"
I20260927 10:57:26.582031 2720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 10:57:26.584276 2735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:26.585402 2720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:26.585528 2720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "32d38f08af674940a7227b997a238338"
format_stamp: "Formatted at 2026-09-27 10:57:26 on dist-test-slave-tz51"
server_key: "430d0c9ebde553c6e4a7806ca5c3c784"
server_key_iv: "3f4b0bae1e15177a743632690496118e"
server_key_version: "encryptionkey@0"
I20260927 10:57:26.585629 2720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:26.605583 2720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:26.608223 2720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:26.608409 2720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:26.615080 2720 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46315
I20260927 10:57:26.615103 2787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46315 every 8 connection(s)
I20260927 10:57:26.615983 2720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:26.616264 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 2720
I20260927 10:57:26.616415 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:26.616737 14608 external_mini_cluster.cc:1428] Setting key 692726b497cf79ecce8daa468fe9edae
I20260927 10:57:26.619035 2788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 10:57:26 dist-test-slave-tz51 krb5kdc[2707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 10:57:26.625506 2788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32d38f08af674940a7227b997a238338: Bootstrap starting.
I20260927 10:57:26.628187 2788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32d38f08af674940a7227b997a238338: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:26.629076 2788 log.cc:824] T 00000000000000000000000000000000 P 32d38f08af674940a7227b997a238338: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:26.630818 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:26.618054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34265 (local address 127.0.0.1:46315)
0927 10:57:26.618482 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:26.618496 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:26.618528 (+ 32us) server_negotiation.cc:407] Connection header received
0927 10:57:26.619163 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:26.619184 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:26.619514 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:26.619835 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:26.620682 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:26.621640 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:26.622817 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:26.623258 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:26.623373 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:26.626065 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:26.626071 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:26.626086 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:26.626116 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:26.627959 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:26.628387 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:26.628393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:26.628398 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:26.628533 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:26.628784 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:26.628787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:26.628788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:26.629270 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:26.629457 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:26.629690 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 10:57:26.629964 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":144,"threads_started":1}
I20260927 10:57:26.631193 2788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32d38f08af674940a7227b997a238338: No bootstrap required, opened a new log
I20260927 10:57:26.633496 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:57:26.634968 2788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32d38f08af674940a7227b997a238338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d38f08af674940a7227b997a238338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } }
I20260927 10:57:26.635196 2788 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32d38f08af674940a7227b997a238338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:26.635289 2788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32d38f08af674940a7227b997a238338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d38f08af674940a7227b997a238338, State: Initialized, Role: FOLLOWER
I20260927 10:57:26.635859 2788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32d38f08af674940a7227b997a238338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d38f08af674940a7227b997a238338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } }
I20260927 10:57:26.636025 2788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32d38f08af674940a7227b997a238338 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:26.636133 2788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32d38f08af674940a7227b997a238338 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:26.636250 2788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32d38f08af674940a7227b997a238338 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:26.637305 2788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32d38f08af674940a7227b997a238338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d38f08af674940a7227b997a238338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } }
I20260927 10:57:26.637642 2788 leader_election.cc:302] T 00000000000000000000000000000000 P 32d38f08af674940a7227b997a238338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32d38f08af674940a7227b997a238338; no voters:
I20260927 10:57:26.637943 2788 leader_election.cc:288] T 00000000000000000000000000000000 P 32d38f08af674940a7227b997a238338 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:26.638041 2798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32d38f08af674940a7227b997a238338 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:26.638249 2798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32d38f08af674940a7227b997a238338 [term 1 LEADER]: Becoming Leader. State: Replica: 32d38f08af674940a7227b997a238338, State: Running, Role: LEADER
I20260927 10:57:26.638604 2798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32d38f08af674940a7227b997a238338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d38f08af674940a7227b997a238338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } }
I20260927 10:57:26.639142 2788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32d38f08af674940a7227b997a238338 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:26.640260 2800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32d38f08af674940a7227b997a238338 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32d38f08af674940a7227b997a238338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d38f08af674940a7227b997a238338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } } }
I20260927 10:57:26.640381 2800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32d38f08af674940a7227b997a238338 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:26.640872 2807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:26.640942 2799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32d38f08af674940a7227b997a238338 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32d38f08af674940a7227b997a238338. Latest consensus state: current_term: 1 leader_uuid: "32d38f08af674940a7227b997a238338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d38f08af674940a7227b997a238338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } } }
I20260927 10:57:26.641433 2799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32d38f08af674940a7227b997a238338 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:26.643882 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:26.635330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54791 (local address 127.0.0.1:46315)
0927 10:57:26.635453 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:26.635457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:26.635828 (+ 371us) server_negotiation.cc:407] Connection header received
0927 10:57:26.635898 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:26.635900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:26.635943 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:26.636039 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:26.636748 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:26.637492 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:26.638494 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:26.638840 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:26.638944 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:26.640562 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:26.640566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:26.640569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:26.640596 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:26.642221 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:26.642629 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:26.642633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:26.642634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:26.642722 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:26.643007 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:26.643010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:26.643011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:26.643202 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:26.643324 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:26.643575 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 10:57:26.643683 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:57:26.644603 2807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:26.649976 2807 catalog_manager.cc:1347] Generated new cluster ID: d6eac494e3664f7ea811907f3716cfb9
I20260927 10:57:26.650053 2807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:26.681679 2807 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:26.682605 2807 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:26.692998 2807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32d38f08af674940a7227b997a238338: Generated new TSK 0
I20260927 10:57:26.693645 2807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:57:26.699156 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 2720
2026-09-27T10:57:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1409 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:26 dist-test-slave-tz51 krb5kdc[2819](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:26 dist-test-slave-tz51 krb5kdc[2819](info): set up 2 sockets
Sep 27 10:57:26 dist-test-slave-tz51 krb5kdc[2819](info): commencing operation
krb5kdc: starting...
W20260927 10:57:27.731945 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:57:27 dist-test-slave-tz51 krb5kdc[2819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:57:27.850813 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:27.960660 2832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:27.960928 2832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:27.961004 2832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:27.964953 2832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:27.965045 2832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:27.965083 2832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:27.965119 2832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:27.965147 2832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:27.970327 2832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:27.971336 2832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:27.972337 2832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:27.980566 2840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:27.980636 2832 server_base.cc:1034] running on GCE node
W20260927 10:57:27.980587 2837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:27.980587 2838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:27.981395 2832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:27.982470 2832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:27.983665 2832 hybrid_clock.cc:648] HybridClock initialized: now 1790506647983651 us; error 26 us; skew 500 ppm
Sep 27 10:57:27 dist-test-slave-tz51 krb5kdc[2819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:27.986536 2832 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:57:27.987432 2832 webserver.cc:460] Webserver started at http://127.0.0.1:38615/ using document root <none> and password file <none>
I20260927 10:57:27.988036 2832 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:27.988124 2832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:27.988350 2832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:27.990345 2832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "f712de6e60a94a708739c61d85705869"
format_stamp: "Formatted at 2026-09-27 10:57:27 on dist-test-slave-tz51"
server_key: "6c0ebcc27d76c358cc0f46104bca2710"
server_key_iv: "2cc0bebb41006295115a39f1f903e871"
server_key_version: "encryptionkey@0"
I20260927 10:57:27.990914 2832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "f712de6e60a94a708739c61d85705869"
format_stamp: "Formatted at 2026-09-27 10:57:27 on dist-test-slave-tz51"
server_key: "6c0ebcc27d76c358cc0f46104bca2710"
server_key_iv: "2cc0bebb41006295115a39f1f903e871"
server_key_version: "encryptionkey@0"
I20260927 10:57:27.994509 2832 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 10:57:27.996870 2847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:27.998087 2832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:57:27.998234 2832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "f712de6e60a94a708739c61d85705869"
format_stamp: "Formatted at 2026-09-27 10:57:27 on dist-test-slave-tz51"
server_key: "6c0ebcc27d76c358cc0f46104bca2710"
server_key_iv: "2cc0bebb41006295115a39f1f903e871"
server_key_version: "encryptionkey@0"
I20260927 10:57:27.998338 2832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:28.015954 2832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:28.019379 2832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:28.019582 2832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:28.026665 2832 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37315
I20260927 10:57:28.026688 2899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37315 every 8 connection(s)
I20260927 10:57:28.027624 2832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:28.030334 2900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:28.035485 2900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f712de6e60a94a708739c61d85705869: Bootstrap starting.
I20260927 10:57:28.036743 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 2832
I20260927 10:57:28.036881 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:28.037182 14608 external_mini_cluster.cc:1428] Setting key 462496e8575ce972e6256c3a61e00d3a
I20260927 10:57:28.038043 2900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f712de6e60a94a708739c61d85705869: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:28.038873 2900 log.cc:824] T 00000000000000000000000000000000 P f712de6e60a94a708739c61d85705869: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:28.040709 2900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f712de6e60a94a708739c61d85705869: No bootstrap required, opened a new log
I20260927 10:57:28.043938 2900 raft_consensus.cc:348] T 00000000000000000000000000000000 P f712de6e60a94a708739c61d85705869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f712de6e60a94a708739c61d85705869" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37315 } }
I20260927 10:57:28.044134 2900 raft_consensus.cc:374] T 00000000000000000000000000000000 P f712de6e60a94a708739c61d85705869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:28.044234 2900 raft_consensus.cc:729] T 00000000000000000000000000000000 P f712de6e60a94a708739c61d85705869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f712de6e60a94a708739c61d85705869, State: Initialized, Role: FOLLOWER
I20260927 10:57:28.044636 2900 consensus_queue.cc:260] T 00000000000000000000000000000000 P f712de6e60a94a708739c61d85705869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f712de6e60a94a708739c61d85705869" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37315 } }
I20260927 10:57:28.044780 2900 raft_consensus.cc:388] T 00000000000000000000000000000000 P f712de6e60a94a708739c61d85705869 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:28.044857 2900 raft_consensus.cc:482] T 00000000000000000000000000000000 P f712de6e60a94a708739c61d85705869 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:28.044955 2900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f712de6e60a94a708739c61d85705869 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:28.045666 2903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:28.038277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50313 (local address 127.0.0.1:37315)
0927 10:57:28.038702 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:28.038717 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:28.038750 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:57:28.039433 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:28.039451 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:28.039852 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:28.040176 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:28.040579 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:28.041559 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:28.042457 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:28.042922 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:28.043099 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:28.043936 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:28.043941 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:28.043953 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:28.044422 (+ 469us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:28.044502 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:28.044534 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:28.044638 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:28.044757 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 10:57:28.044957 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":123,"threads_started":1}
I20260927 10:57:28.045794 2900 raft_consensus.cc:504] T 00000000000000000000000000000000 P f712de6e60a94a708739c61d85705869 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f712de6e60a94a708739c61d85705869" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37315 } }
I20260927 10:57:28.046180 2900 leader_election.cc:302] T 00000000000000000000000000000000 P f712de6e60a94a708739c61d85705869 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f712de6e60a94a708739c61d85705869; no voters:
I20260927 10:57:28.046487 2900 leader_election.cc:288] T 00000000000000000000000000000000 P f712de6e60a94a708739c61d85705869 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:28.046603 2905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f712de6e60a94a708739c61d85705869 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:28.046804 2905 raft_consensus.cc:686] T 00000000000000000000000000000000 P f712de6e60a94a708739c61d85705869 [term 1 LEADER]: Becoming Leader. State: Replica: f712de6e60a94a708739c61d85705869, State: Running, Role: LEADER
I20260927 10:57:28.047111 2905 consensus_queue.cc:237] T 00000000000000000000000000000000 P f712de6e60a94a708739c61d85705869 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f712de6e60a94a708739c61d85705869" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37315 } }
I20260927 10:57:28.047638 2900 sys_catalog.cc:564] T 00000000000000000000000000000000 P f712de6e60a94a708739c61d85705869 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:28.048632 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:57:28.048856 2907 sys_catalog.cc:455] T 00000000000000000000000000000000 P f712de6e60a94a708739c61d85705869 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f712de6e60a94a708739c61d85705869. Latest consensus state: current_term: 1 leader_uuid: "f712de6e60a94a708739c61d85705869" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f712de6e60a94a708739c61d85705869" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37315 } } }
I20260927 10:57:28.048969 2907 sys_catalog.cc:458] T 00000000000000000000000000000000 P f712de6e60a94a708739c61d85705869 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:28.049235 2906 sys_catalog.cc:455] T 00000000000000000000000000000000 P f712de6e60a94a708739c61d85705869 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f712de6e60a94a708739c61d85705869" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f712de6e60a94a708739c61d85705869" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37315 } } }
I20260927 10:57:28.049322 2906 sys_catalog.cc:458] T 00000000000000000000000000000000 P f712de6e60a94a708739c61d85705869 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:28.049330 2914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:28.052435 2914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:28.056720 2903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:28.050679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49655 (local address 127.0.0.1:37315)
0927 10:57:28.050772 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:28.050776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:28.051357 (+ 581us) server_negotiation.cc:407] Connection header received
0927 10:57:28.051629 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:28.051632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:28.051677 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:28.051784 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:28.052026 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:28.052792 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:28.053673 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:28.053913 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:28.054028 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:28.056260 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:28.056263 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:28.056266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:28.056277 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:28.056310 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:28.056320 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:28.056397 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:28.056522 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 10:57:28.056567 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 10:57:28.058045 2914 catalog_manager.cc:1347] Generated new cluster ID: 9883fff6c50b4858aa2540f015971ccb
I20260927 10:57:28.058112 2914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:28.070924 2914 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:28.072149 2914 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:28.080353 2914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f712de6e60a94a708739c61d85705869: Generated new TSK 0
I20260927 10:57:28.081051 2914 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:57:28.088344 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 2832
2026-09-27T10:57:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:28 dist-test-slave-tz51 krb5kdc[2931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:28 dist-test-slave-tz51 krb5kdc[2931](info): set up 2 sockets
Sep 27 10:57:28 dist-test-slave-tz51 krb5kdc[2931](info): commencing operation
krb5kdc: starting...
W20260927 10:57:29.124090 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:57:29 dist-test-slave-tz51 krb5kdc[2931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:57:29.238796 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:29.345925 2944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:29.346290 2944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:29.346381 2944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:29.350461 2944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:29.350553 2944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:29.350591 2944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:29.350639 2944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:29.350678 2944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:29.356637 2944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:29.357674 2944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:29.358669 2944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:29.367503 2950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:29.367561 2944 server_base.cc:1034] running on GCE node
W20260927 10:57:29.367503 2949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:29.367745 2952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:29.368284 2944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:29.369350 2944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:29.370529 2944 hybrid_clock.cc:648] HybridClock initialized: now 1790506649370515 us; error 30 us; skew 500 ppm
Sep 27 10:57:29 dist-test-slave-tz51 krb5kdc[2931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:29.373190 2944 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:57:29.374056 2944 webserver.cc:460] Webserver started at http://127.0.0.1:41171/ using document root <none> and password file <none>
I20260927 10:57:29.374547 2944 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:29.374603 2944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:29.374760 2944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:29.376704 2944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "32e47c026ea94d9fb7905163b008f64d"
format_stamp: "Formatted at 2026-09-27 10:57:29 on dist-test-slave-tz51"
server_key: "973730d631e401dd626b91538a8d945d"
server_key_iv: "cc55c37567c296a8d55a750080d2441a"
server_key_version: "encryptionkey@0"
I20260927 10:57:29.377192 2944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "32e47c026ea94d9fb7905163b008f64d"
format_stamp: "Formatted at 2026-09-27 10:57:29 on dist-test-slave-tz51"
server_key: "973730d631e401dd626b91538a8d945d"
server_key_iv: "cc55c37567c296a8d55a750080d2441a"
server_key_version: "encryptionkey@0"
I20260927 10:57:29.380652 2944 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:57:29.382848 2959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:29.384126 2944 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:57:29.384230 2944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "32e47c026ea94d9fb7905163b008f64d"
format_stamp: "Formatted at 2026-09-27 10:57:29 on dist-test-slave-tz51"
server_key: "973730d631e401dd626b91538a8d945d"
server_key_iv: "cc55c37567c296a8d55a750080d2441a"
server_key_version: "encryptionkey@0"
I20260927 10:57:29.384308 2944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:29.388528 2944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:29.391479 2944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:29.391614 2944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:29.398460 2944 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36305
I20260927 10:57:29.398487 3011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36305 every 8 connection(s)
I20260927 10:57:29.399320 2944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:29.401890 3012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:29.404101 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 2944
I20260927 10:57:29.404239 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:29.404493 14608 external_mini_cluster.cc:1428] Setting key bd1d1afc1bce2bf74841bb79a0a7be77
I20260927 10:57:29.408039 3012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32e47c026ea94d9fb7905163b008f64d: Bootstrap starting.
I20260927 10:57:29.409896 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:29.405806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45011 (local address 127.0.0.1:36305)
0927 10:57:29.406229 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:29.406248 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:29.406290 (+ 42us) server_negotiation.cc:407] Connection header received
0927 10:57:29.407136 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:29.407159 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:29.407496 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:29.407866 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:29.408568 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:29.408574 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:29.408589 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:29.408601 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:29.408664 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:29.408690 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:29.408818 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:29.408881 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 10:57:29.409020 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":121,"threads_started":1}
I20260927 10:57:29.410615 3012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32e47c026ea94d9fb7905163b008f64d: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:29.411310 3012 log.cc:824] T 00000000000000000000000000000000 P 32e47c026ea94d9fb7905163b008f64d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:29.412474 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:57:29.413169 3012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32e47c026ea94d9fb7905163b008f64d: No bootstrap required, opened a new log
I20260927 10:57:29.415863 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:29.414448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58617 (local address 127.0.0.1:36305)
0927 10:57:29.414547 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:29.414552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:29.414678 (+ 126us) server_negotiation.cc:407] Connection header received
0927 10:57:29.414771 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:29.414774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:29.414805 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:29.414895 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:29.415493 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:29.415499 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:29.415504 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:29.415515 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:29.415546 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:29.415559 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:29.415651 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:29.415683 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 10:57:29.415738 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:57:29.416987 3012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32e47c026ea94d9fb7905163b008f64d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e47c026ea94d9fb7905163b008f64d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36305 } }
I20260927 10:57:29.417184 3012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32e47c026ea94d9fb7905163b008f64d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:29.417217 3012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32e47c026ea94d9fb7905163b008f64d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e47c026ea94d9fb7905163b008f64d, State: Initialized, Role: FOLLOWER
I20260927 10:57:29.417553 3012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32e47c026ea94d9fb7905163b008f64d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e47c026ea94d9fb7905163b008f64d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36305 } }
I20260927 10:57:29.417659 3012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32e47c026ea94d9fb7905163b008f64d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:29.417702 3012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32e47c026ea94d9fb7905163b008f64d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:29.417770 3012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32e47c026ea94d9fb7905163b008f64d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:29.418514 3012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32e47c026ea94d9fb7905163b008f64d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e47c026ea94d9fb7905163b008f64d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36305 } }
I20260927 10:57:29.418797 3012 leader_election.cc:302] T 00000000000000000000000000000000 P 32e47c026ea94d9fb7905163b008f64d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32e47c026ea94d9fb7905163b008f64d; no voters:
I20260927 10:57:29.419014 3012 leader_election.cc:288] T 00000000000000000000000000000000 P 32e47c026ea94d9fb7905163b008f64d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:29.419091 3022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32e47c026ea94d9fb7905163b008f64d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:29.419325 3022 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32e47c026ea94d9fb7905163b008f64d [term 1 LEADER]: Becoming Leader. State: Replica: 32e47c026ea94d9fb7905163b008f64d, State: Running, Role: LEADER
I20260927 10:57:29.419647 3022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32e47c026ea94d9fb7905163b008f64d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e47c026ea94d9fb7905163b008f64d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36305 } }
I20260927 10:57:29.420202 3012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32e47c026ea94d9fb7905163b008f64d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:29.421355 3023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32e47c026ea94d9fb7905163b008f64d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32e47c026ea94d9fb7905163b008f64d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e47c026ea94d9fb7905163b008f64d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36305 } } }
I20260927 10:57:29.421332 3024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32e47c026ea94d9fb7905163b008f64d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32e47c026ea94d9fb7905163b008f64d. Latest consensus state: current_term: 1 leader_uuid: "32e47c026ea94d9fb7905163b008f64d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e47c026ea94d9fb7905163b008f64d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36305 } } }
I20260927 10:57:29.421464 3024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32e47c026ea94d9fb7905163b008f64d [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:29.421464 3023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32e47c026ea94d9fb7905163b008f64d [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:29.421873 3031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:29.424889 3031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:29.429940 3031 catalog_manager.cc:1347] Generated new cluster ID: d477af35a9d4473f964201a385170068
I20260927 10:57:29.430004 3031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:29.454880 3031 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:29.455228 3031 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:29.459779 3031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32e47c026ea94d9fb7905163b008f64d: Generated new TSK 0
I20260927 10:57:29.460618 3031 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:57:29.473582 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 2944
2026-09-27T10:57:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:29 dist-test-slave-tz51 krb5kdc[3043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:29 dist-test-slave-tz51 krb5kdc[3043](info): set up 2 sockets
Sep 27 10:57:29 dist-test-slave-tz51 krb5kdc[3043](info): commencing operation
krb5kdc: starting...
W20260927 10:57:30.503942 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:57:30 dist-test-slave-tz51 krb5kdc[3043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:57:30.619146 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41023
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:30.726130 3056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:30.726367 3056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:30.726440 3056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:30.730489 3056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:30.730538 3056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:30.730558 3056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:30.730608 3056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:30.730640 3056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:30.735741 3056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:30.736711 3056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:30.737697 3056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:30.745970 3064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:30.746090 3056 server_base.cc:1034] running on GCE node
W20260927 10:57:30.746163 3062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:30.745978 3061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:30.746742 3056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:30.747846 3056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:30.749040 3056 hybrid_clock.cc:648] HybridClock initialized: now 1790506650749023 us; error 29 us; skew 500 ppm
Sep 27 10:57:30 dist-test-slave-tz51 krb5kdc[3043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:30.756376 3056 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:57:30.757843 3056 webserver.cc:460] Webserver started at http://127.0.0.1:44799/ using document root <none> and password file <none>
I20260927 10:57:30.758821 3056 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:30.758915 3056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:30.759214 3056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:30.762272 3056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "3a30bfceb10d475382e3ff983ea5a857"
format_stamp: "Formatted at 2026-09-27 10:57:30 on dist-test-slave-tz51"
server_key: "4e1779ceeb7be4470c612876f90d45d8"
server_key_iv: "85baa7f4bbce6c7ac397608f48ac3d50"
server_key_version: "encryptionkey@0"
I20260927 10:57:30.763164 3056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "3a30bfceb10d475382e3ff983ea5a857"
format_stamp: "Formatted at 2026-09-27 10:57:30 on dist-test-slave-tz51"
server_key: "4e1779ceeb7be4470c612876f90d45d8"
server_key_iv: "85baa7f4bbce6c7ac397608f48ac3d50"
server_key_version: "encryptionkey@0"
I20260927 10:57:30.768980 3056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 10:57:30.772015 3071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:30.773576 3056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:57:30.773718 3056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "3a30bfceb10d475382e3ff983ea5a857"
format_stamp: "Formatted at 2026-09-27 10:57:30 on dist-test-slave-tz51"
server_key: "4e1779ceeb7be4470c612876f90d45d8"
server_key_iv: "85baa7f4bbce6c7ac397608f48ac3d50"
server_key_version: "encryptionkey@0"
I20260927 10:57:30.773844 3056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:30.793154 3056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:30.796319 3056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:30.796504 3056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:30.803820 3056 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41023
I20260927 10:57:30.803835 3123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41023 every 8 connection(s)
I20260927 10:57:30.804777 3056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:30.807441 3124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:30.814273 3124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857: Bootstrap starting.
I20260927 10:57:30.814872 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 3056
I20260927 10:57:30.815011 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:30.815366 14608 external_mini_cluster.cc:1428] Setting key 643d53e4c151ce6d264b025cd3276ff2
I20260927 10:57:30.817226 3124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:30.818041 3124 log.cc:824] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:30.819936 3124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857: No bootstrap required, opened a new log
I20260927 10:57:30.823143 3124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a30bfceb10d475382e3ff983ea5a857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41023 } }
I20260927 10:57:30.823382 3124 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:30.823453 3124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a30bfceb10d475382e3ff983ea5a857, State: Initialized, Role: FOLLOWER
I20260927 10:57:30.823912 3124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a30bfceb10d475382e3ff983ea5a857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41023 } }
I20260927 10:57:30.824071 3124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:30.824146 3124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:30.824247 3124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 10:57:30 dist-test-slave-tz51 krb5kdc[3043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 10:57:30.825183 3124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a30bfceb10d475382e3ff983ea5a857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41023 } }
I20260927 10:57:30.825500 3124 leader_election.cc:302] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a30bfceb10d475382e3ff983ea5a857; no voters:
I20260927 10:57:30.825766 3124 leader_election.cc:288] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:30.825875 3129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:30.826151 3129 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857 [term 1 LEADER]: Becoming Leader. State: Replica: 3a30bfceb10d475382e3ff983ea5a857, State: Running, Role: LEADER
I20260927 10:57:30.826455 3129 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a30bfceb10d475382e3ff983ea5a857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41023 } }
I20260927 10:57:30.826926 3124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:30.828400 3130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a30bfceb10d475382e3ff983ea5a857" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a30bfceb10d475382e3ff983ea5a857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41023 } } }
I20260927 10:57:30.828537 3130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:30.830133 3131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a30bfceb10d475382e3ff983ea5a857. Latest consensus state: current_term: 1 leader_uuid: "3a30bfceb10d475382e3ff983ea5a857" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a30bfceb10d475382e3ff983ea5a857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41023 } } }
I20260927 10:57:30.830236 3131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857 [sys.catalog]: This master's current role is: LEADER
W20260927 10:57:30.832453 3145 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:57:30.832521 3145 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:57:30.832587 3141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:30.833173 3127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:30.816709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:49937 (local address 127.0.0.1:41023)
0927 10:57:30.817159 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:30.817177 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:30.817219 (+ 42us) server_negotiation.cc:407] Connection header received
0927 10:57:30.818196 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:30.818220 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:30.818634 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:30.819015 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:30.819845 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:30.820855 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:30.821721 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:30.822213 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:30.822374 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:30.825210 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:30.825220 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:30.825232 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:30.825276 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:30.829876 (+ 4600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:30.830508 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:30.830514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:30.830520 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:30.830661 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:30.831089 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:30.831092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:30.831094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:30.831565 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:30.831801 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:30.832127 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 10:57:30.832348 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":121,"threads_started":1}
I20260927 10:57:30.833295 3141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:30.835944 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:57:30.838660 3141 catalog_manager.cc:1347] Generated new cluster ID: 1669fcf9ccc04412a72b44732d413ca8
I20260927 10:57:30.838721 3141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:30.847299 3127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:30.838024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:34491 (local address 127.0.0.1:41023)
0927 10:57:30.838160 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:30.838164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:30.839954 (+ 1790us) server_negotiation.cc:407] Connection header received
0927 10:57:30.840095 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:30.840098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:30.840149 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:30.840225 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:30.840861 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:30.841371 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:30.842196 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:30.842443 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:30.842543 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:30.844040 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:30.844043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:30.844046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:30.844069 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:30.845442 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:30.845935 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:30.845938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:30.845940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:30.846025 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:30.846344 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:30.846348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:30.846349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:30.846516 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:30.846665 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:30.846949 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 10:57:30.847040 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:57:30.853291 3141 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:30.854507 3141 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:30.858656 3141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a30bfceb10d475382e3ff983ea5a857: Generated new TSK 0
I20260927 10:57:30.859282 3141 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:57:30.862833 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 3056
W20260927 10:57:30.868466 3041 connection.cc:535] client connection to 127.0.0.1:41023 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41023): Connection reset by peer (error 104)
2026-09-27T10:57:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:30 dist-test-slave-tz51 krb5kdc[3155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:30 dist-test-slave-tz51 krb5kdc[3155](info): set up 2 sockets
Sep 27 10:57:30 dist-test-slave-tz51 krb5kdc[3155](info): commencing operation
krb5kdc: starting...
W20260927 10:57:31.891918 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:57:31 dist-test-slave-tz51 krb5kdc[3155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:57:32.008354 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37463
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:32.118343 3168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:32.118604 3168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:32.118646 3168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:32.122617 3168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:32.122677 3168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:32.122716 3168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:32.122756 3168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:32.122784 3168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:32.127990 3168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:32.128925 3168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:32.129855 3168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:32.138041 3176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:32.138016 3173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:32.138016 3174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:32.138239 3168 server_base.cc:1034] running on GCE node
I20260927 10:57:32.138775 3168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:32.139840 3168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:32.141009 3168 hybrid_clock.cc:648] HybridClock initialized: now 1790506652140985 us; error 40 us; skew 500 ppm
Sep 27 10:57:32 dist-test-slave-tz51 krb5kdc[3155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:32.149159 3168 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:57:32.150007 3168 webserver.cc:460] Webserver started at http://127.0.0.1:45445/ using document root <none> and password file <none>
I20260927 10:57:32.150535 3168 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:32.150621 3168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:32.150840 3168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:32.152809 3168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "cc2b9bff4a37457ba09db3ca58d64db0"
format_stamp: "Formatted at 2026-09-27 10:57:32 on dist-test-slave-tz51"
server_key: "8cc124ca19709ab637df229b02bd4e5f"
server_key_iv: "efd691caf39108531a822c58145d0614"
server_key_version: "encryptionkey@0"
I20260927 10:57:32.153358 3168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "cc2b9bff4a37457ba09db3ca58d64db0"
format_stamp: "Formatted at 2026-09-27 10:57:32 on dist-test-slave-tz51"
server_key: "8cc124ca19709ab637df229b02bd4e5f"
server_key_iv: "efd691caf39108531a822c58145d0614"
server_key_version: "encryptionkey@0"
I20260927 10:57:32.156773 3168 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:57:32.159025 3183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:32.160238 3168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:57:32.160382 3168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "cc2b9bff4a37457ba09db3ca58d64db0"
format_stamp: "Formatted at 2026-09-27 10:57:32 on dist-test-slave-tz51"
server_key: "8cc124ca19709ab637df229b02bd4e5f"
server_key_iv: "efd691caf39108531a822c58145d0614"
server_key_version: "encryptionkey@0"
I20260927 10:57:32.160478 3168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:32.169556 3168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:32.172183 3168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:32.172360 3168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:32.178987 3168 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37463
I20260927 10:57:32.179013 3235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37463 every 8 connection(s)
I20260927 10:57:32.179986 3168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:32.182695 3236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:32.183758 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 3168
I20260927 10:57:32.183908 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:32.184211 14608 external_mini_cluster.cc:1428] Setting key a6eb0ee0335ab09c1df508b128976475
I20260927 10:57:32.189289 3236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0: Bootstrap starting.
I20260927 10:57:32.191808 3236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:32.192672 3236 log.cc:824] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:32.193894 3239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:32.185540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:41499 (local address 127.0.0.1:37463)
0927 10:57:32.185985 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:32.186018 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:32.186061 (+ 43us) server_negotiation.cc:407] Connection header received
0927 10:57:32.186874 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:32.186896 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:32.187320 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:32.187683 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:32.188126 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:32.189557 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:32.190423 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:32.191086 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:32.191246 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:32.192127 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:32.192133 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:32.192148 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:32.192626 (+ 478us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:32.192704 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:32.192736 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:32.192875 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:32.193012 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 10:57:32.193144 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":121,"threads_started":1}
I20260927 10:57:32.194571 3236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0: No bootstrap required, opened a new log
I20260927 10:57:32.196594 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:57:32.198199 3236 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2b9bff4a37457ba09db3ca58d64db0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37463 } }
I20260927 10:57:32.198468 3236 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:32.198535 3236 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc2b9bff4a37457ba09db3ca58d64db0, State: Initialized, Role: FOLLOWER
I20260927 10:57:32.199002 3236 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2b9bff4a37457ba09db3ca58d64db0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37463 } }
I20260927 10:57:32.199182 3236 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:32.199265 3236 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:32.199370 3236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:32.200352 3236 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2b9bff4a37457ba09db3ca58d64db0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37463 } }
I20260927 10:57:32.200668 3236 leader_election.cc:302] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc2b9bff4a37457ba09db3ca58d64db0; no voters:
I20260927 10:57:32.200968 3236 leader_election.cc:288] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:32.201030 3246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:32.201227 3246 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0 [term 1 LEADER]: Becoming Leader. State: Replica: cc2b9bff4a37457ba09db3ca58d64db0, State: Running, Role: LEADER
I20260927 10:57:32.201515 3246 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2b9bff4a37457ba09db3ca58d64db0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37463 } }
I20260927 10:57:32.202242 3236 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:32.202740 3239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:32.198414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:46243 (local address 127.0.0.1:37463)
0927 10:57:32.198525 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:32.198528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:32.198774 (+ 246us) server_negotiation.cc:407] Connection header received
0927 10:57:32.198869 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:32.198877 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:32.198912 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:32.198973 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:32.199265 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:32.199858 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:32.200897 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:32.201177 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:32.201282 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:32.202236 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:32.202240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:32.202242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:32.202254 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:32.202280 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:32.202290 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:32.202384 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:32.202561 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 10:57:32.202594 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:57:32.202908 3246 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc2b9bff4a37457ba09db3ca58d64db0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2b9bff4a37457ba09db3ca58d64db0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37463 } } }
I20260927 10:57:32.203040 3246 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:32.203483 3247 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc2b9bff4a37457ba09db3ca58d64db0. Latest consensus state: current_term: 1 leader_uuid: "cc2b9bff4a37457ba09db3ca58d64db0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2b9bff4a37457ba09db3ca58d64db0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37463 } } }
I20260927 10:57:32.203732 3247 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0 [sys.catalog]: This master's current role is: LEADER
W20260927 10:57:32.206602 3261 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:57:32.206686 3261 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:57:32.206766 3254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:32.207412 3254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:32.211974 3254 catalog_manager.cc:1347] Generated new cluster ID: b44c4346cc6a45ae9c26be7ae1a66281
I20260927 10:57:32.212034 3254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:32.220073 3254 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:32.220852 3254 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:32.229101 3254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc2b9bff4a37457ba09db3ca58d64db0: Generated new TSK 0
I20260927 10:57:32.229672 3254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:57:32.240141 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 3168
W20260927 10:57:32.245584 3153 connection.cc:535] client connection to 127.0.0.1:37463 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37463): Connection reset by peer (error 104)
2026-09-27T10:57:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:32 dist-test-slave-tz51 krb5kdc[3267](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:32 dist-test-slave-tz51 krb5kdc[3267](info): set up 2 sockets
Sep 27 10:57:32 dist-test-slave-tz51 krb5kdc[3267](info): commencing operation
krb5kdc: starting...
W20260927 10:57:33.288066 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:57:33 dist-test-slave-tz51 krb5kdc[3267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:57:33.406253 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43987
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:33.513897 3280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:33.514154 3280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:33.514195 3280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:33.518193 3280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:33.518246 3280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:33.518266 3280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:33.518282 3280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:33.518335 3280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:33.523514 3280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:33.524534 3280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:33.525530 3280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:33.533623 3288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:33.533623 3285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:33.533603 3286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:33.533825 3280 server_base.cc:1034] running on GCE node
I20260927 10:57:33.534411 3280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:33.535485 3280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:33.536664 3280 hybrid_clock.cc:648] HybridClock initialized: now 1790506653536642 us; error 36 us; skew 500 ppm
Sep 27 10:57:33 dist-test-slave-tz51 krb5kdc[3267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:33.539414 3280 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:57:33.540303 3280 webserver.cc:460] Webserver started at http://127.0.0.1:39711/ using document root <none> and password file <none>
I20260927 10:57:33.540832 3280 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:33.540916 3280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:33.541144 3280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:33.543108 3280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "83217ff24adc4622b9a1cccfd8f29b13"
format_stamp: "Formatted at 2026-09-27 10:57:33 on dist-test-slave-tz51"
server_key: "117652350ded22658712de05fc39ef65"
server_key_iv: "16746e3fa17cac04ee641f5994ee7f2c"
server_key_version: "encryptionkey@0"
I20260927 10:57:33.543654 3280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "83217ff24adc4622b9a1cccfd8f29b13"
format_stamp: "Formatted at 2026-09-27 10:57:33 on dist-test-slave-tz51"
server_key: "117652350ded22658712de05fc39ef65"
server_key_iv: "16746e3fa17cac04ee641f5994ee7f2c"
server_key_version: "encryptionkey@0"
I20260927 10:57:33.547312 3280 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:57:33.549832 3295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:33.551111 3280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:33.551263 3280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "83217ff24adc4622b9a1cccfd8f29b13"
format_stamp: "Formatted at 2026-09-27 10:57:33 on dist-test-slave-tz51"
server_key: "117652350ded22658712de05fc39ef65"
server_key_iv: "16746e3fa17cac04ee641f5994ee7f2c"
server_key_version: "encryptionkey@0"
I20260927 10:57:33.551383 3280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:33.556151 3280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:33.559470 3280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:33.559657 3280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:33.566424 3280 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43987
I20260927 10:57:33.566434 3347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43987 every 8 connection(s)
I20260927 10:57:33.567364 3280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:33.570060 3348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:33.571532 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 3280
I20260927 10:57:33.571671 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:33.572010 14608 external_mini_cluster.cc:1428] Setting key 3b5c781f27c7084fad38f42fd613c54f
I20260927 10:57:33.576156 3348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83217ff24adc4622b9a1cccfd8f29b13: Bootstrap starting.
I20260927 10:57:33.577114 3351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:33.573497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:48437 (local address 127.0.0.1:43987)
0927 10:57:33.573921 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:33.573935 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:33.573971 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:57:33.574620 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:33.574636 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:33.574901 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:33.575237 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:33.575989 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:33.575994 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:33.576013 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:33.576023 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:33.576073 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:33.576092 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:33.576236 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:33.576307 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 10:57:33.576456 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":126,"threads_started":1}
I20260927 10:57:33.578516 3348 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83217ff24adc4622b9a1cccfd8f29b13: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:33.579355 3348 log.cc:824] T 00000000000000000000000000000000 P 83217ff24adc4622b9a1cccfd8f29b13: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:33.579787 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:57:33.581149 3348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83217ff24adc4622b9a1cccfd8f29b13: No bootstrap required, opened a new log
I20260927 10:57:33.583235 3351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:33.581741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:39377 (local address 127.0.0.1:43987)
0927 10:57:33.581858 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:33.581862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:33.582038 (+ 176us) server_negotiation.cc:407] Connection header received
0927 10:57:33.582166 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:33.582168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:33.582193 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:33.582254 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:33.582883 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:33.582885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:33.582888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:33.582897 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:33.582920 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:33.582929 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:33.583067 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:33.583090 (+ 23us) server_negotiation.cc:299] Negotiation successful
0927 10:57:33.583119 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:57:33.585316 3348 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83217ff24adc4622b9a1cccfd8f29b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83217ff24adc4622b9a1cccfd8f29b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43987 } }
I20260927 10:57:33.585532 3348 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83217ff24adc4622b9a1cccfd8f29b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:33.585587 3348 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83217ff24adc4622b9a1cccfd8f29b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83217ff24adc4622b9a1cccfd8f29b13, State: Initialized, Role: FOLLOWER
I20260927 10:57:33.586014 3348 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83217ff24adc4622b9a1cccfd8f29b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83217ff24adc4622b9a1cccfd8f29b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43987 } }
I20260927 10:57:33.586155 3348 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83217ff24adc4622b9a1cccfd8f29b13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:33.586220 3348 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83217ff24adc4622b9a1cccfd8f29b13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:33.586310 3348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83217ff24adc4622b9a1cccfd8f29b13 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:33.587105 3348 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83217ff24adc4622b9a1cccfd8f29b13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83217ff24adc4622b9a1cccfd8f29b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43987 } }
I20260927 10:57:33.587415 3348 leader_election.cc:302] T 00000000000000000000000000000000 P 83217ff24adc4622b9a1cccfd8f29b13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83217ff24adc4622b9a1cccfd8f29b13; no voters:
I20260927 10:57:33.587728 3348 leader_election.cc:288] T 00000000000000000000000000000000 P 83217ff24adc4622b9a1cccfd8f29b13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:33.587841 3358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83217ff24adc4622b9a1cccfd8f29b13 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:33.588071 3358 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83217ff24adc4622b9a1cccfd8f29b13 [term 1 LEADER]: Becoming Leader. State: Replica: 83217ff24adc4622b9a1cccfd8f29b13, State: Running, Role: LEADER
I20260927 10:57:33.588377 3358 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83217ff24adc4622b9a1cccfd8f29b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83217ff24adc4622b9a1cccfd8f29b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43987 } }
I20260927 10:57:33.588869 3348 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83217ff24adc4622b9a1cccfd8f29b13 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:33.589757 3360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83217ff24adc4622b9a1cccfd8f29b13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83217ff24adc4622b9a1cccfd8f29b13. Latest consensus state: current_term: 1 leader_uuid: "83217ff24adc4622b9a1cccfd8f29b13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83217ff24adc4622b9a1cccfd8f29b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43987 } } }
I20260927 10:57:33.589885 3360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83217ff24adc4622b9a1cccfd8f29b13 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:33.590193 3359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83217ff24adc4622b9a1cccfd8f29b13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83217ff24adc4622b9a1cccfd8f29b13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83217ff24adc4622b9a1cccfd8f29b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43987 } } }
I20260927 10:57:33.590287 3359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83217ff24adc4622b9a1cccfd8f29b13 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:33.590569 3367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:33.593513 3367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:33.598289 3367 catalog_manager.cc:1347] Generated new cluster ID: 5b66fa7aaccc4b3eb7c8bb56d08dae49
I20260927 10:57:33.598349 3367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:33.630532 3367 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:33.630862 3367 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:33.641134 3367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83217ff24adc4622b9a1cccfd8f29b13: Generated new TSK 0
I20260927 10:57:33.641780 3367 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:57:33.647436 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 3280
2026-09-27T10:57:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:33 dist-test-slave-tz51 krb5kdc[3379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:33 dist-test-slave-tz51 krb5kdc[3379](info): set up 2 sockets
Sep 27 10:57:33 dist-test-slave-tz51 krb5kdc[3379](info): commencing operation
krb5kdc: starting...
W20260927 10:57:34.683866 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:57:34 dist-test-slave-tz51 krb5kdc[3379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:57:34.802479 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37243
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:34.913952 3392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:34.914234 3392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:34.914310 3392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:34.918244 3392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:34.918321 3392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:34.918360 3392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:34.918411 3392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:34.918450 3392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:34.923489 3392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37243
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:34.924465 3392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:34.925420 3392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:34.933578 3397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:34.933597 3400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:34.933593 3398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:34.933756 3392 server_base.cc:1034] running on GCE node
I20260927 10:57:34.934358 3392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:34.935396 3392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:34.936575 3392 hybrid_clock.cc:648] HybridClock initialized: now 1790506654936545 us; error 55 us; skew 500 ppm
Sep 27 10:57:34 dist-test-slave-tz51 krb5kdc[3379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:34.939189 3392 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:57:34.940068 3392 webserver.cc:460] Webserver started at http://127.0.0.1:34887/ using document root <none> and password file <none>
I20260927 10:57:34.940589 3392 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:34.940673 3392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:34.940879 3392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:34.942803 3392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "4ed56951cbe4419781dfd0c9ceede07b"
format_stamp: "Formatted at 2026-09-27 10:57:34 on dist-test-slave-tz51"
server_key: "1431c4d5e1c957da6e7a03dc2bb5134e"
server_key_iv: "03bf5f501c59b10be79cbc8d9f8506ae"
server_key_version: "encryptionkey@0"
I20260927 10:57:34.943356 3392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "4ed56951cbe4419781dfd0c9ceede07b"
format_stamp: "Formatted at 2026-09-27 10:57:34 on dist-test-slave-tz51"
server_key: "1431c4d5e1c957da6e7a03dc2bb5134e"
server_key_iv: "03bf5f501c59b10be79cbc8d9f8506ae"
server_key_version: "encryptionkey@0"
I20260927 10:57:34.947026 3392 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:57:34.949374 3407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:34.950583 3392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 10:57:34.950708 3392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "4ed56951cbe4419781dfd0c9ceede07b"
format_stamp: "Formatted at 2026-09-27 10:57:34 on dist-test-slave-tz51"
server_key: "1431c4d5e1c957da6e7a03dc2bb5134e"
server_key_iv: "03bf5f501c59b10be79cbc8d9f8506ae"
server_key_version: "encryptionkey@0"
I20260927 10:57:34.950809 3392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:34.972153 3392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:34.975219 3392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:34.975406 3392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:34.981971 3392 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37243
I20260927 10:57:34.982002 3459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37243 every 8 connection(s)
I20260927 10:57:34.982853 3392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:34.985484 3460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:34.988582 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 3392
I20260927 10:57:34.988713 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:34.988976 14608 external_mini_cluster.cc:1428] Setting key 3e1beeffcbe37df0445029f6019f3964
I20260927 10:57:34.991191 3460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ed56951cbe4419781dfd0c9ceede07b: Bootstrap starting.
I20260927 10:57:34.993561 3460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ed56951cbe4419781dfd0c9ceede07b: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:34.994329 3460 log.cc:824] T 00000000000000000000000000000000 P 4ed56951cbe4419781dfd0c9ceede07b: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:34.996150 3460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ed56951cbe4419781dfd0c9ceede07b: No bootstrap required, opened a new log
Sep 27 10:57:34 dist-test-slave-tz51 krb5kdc[3379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 10:57:34.999832 3460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ed56951cbe4419781dfd0c9ceede07b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed56951cbe4419781dfd0c9ceede07b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37243 } }
I20260927 10:57:35.000059 3460 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ed56951cbe4419781dfd0c9ceede07b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:35.000113 3460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ed56951cbe4419781dfd0c9ceede07b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed56951cbe4419781dfd0c9ceede07b, State: Initialized, Role: FOLLOWER
I20260927 10:57:35.000523 3460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ed56951cbe4419781dfd0c9ceede07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed56951cbe4419781dfd0c9ceede07b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37243 } }
I20260927 10:57:35.000676 3460 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ed56951cbe4419781dfd0c9ceede07b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:35.000757 3460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ed56951cbe4419781dfd0c9ceede07b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:35.000876 3460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ed56951cbe4419781dfd0c9ceede07b [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:35.001680 3460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ed56951cbe4419781dfd0c9ceede07b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed56951cbe4419781dfd0c9ceede07b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37243 } }
I20260927 10:57:35.001998 3460 leader_election.cc:302] T 00000000000000000000000000000000 P 4ed56951cbe4419781dfd0c9ceede07b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ed56951cbe4419781dfd0c9ceede07b; no voters:
I20260927 10:57:35.002288 3460 leader_election.cc:288] T 00000000000000000000000000000000 P 4ed56951cbe4419781dfd0c9ceede07b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:35.002393 3465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ed56951cbe4419781dfd0c9ceede07b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:35.002678 3465 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ed56951cbe4419781dfd0c9ceede07b [term 1 LEADER]: Becoming Leader. State: Replica: 4ed56951cbe4419781dfd0c9ceede07b, State: Running, Role: LEADER
I20260927 10:57:35.002985 3465 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ed56951cbe4419781dfd0c9ceede07b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed56951cbe4419781dfd0c9ceede07b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37243 } }
I20260927 10:57:35.003512 3460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ed56951cbe4419781dfd0c9ceede07b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:35.003774 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:34.990315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:54491 (local address 127.0.0.1:37243)
0927 10:57:34.990755 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:34.990769 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:34.990803 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:57:34.991575 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:34.991605 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:34.991996 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:34.992517 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:34.993349 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:34.994380 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:34.995269 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:34.995789 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:34.995945 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:34.998685 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:34.998690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:34.998703 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:34.998729 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:35.000544 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:35.001154 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:35.001165 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:35.001169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:35.001273 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:35.001602 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:35.001605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:35.001607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:35.002018 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:35.002216 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:35.002604 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 10:57:35.002816 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":139,"threads_started":1}
I20260927 10:57:35.004083 3465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ed56951cbe4419781dfd0c9ceede07b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ed56951cbe4419781dfd0c9ceede07b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed56951cbe4419781dfd0c9ceede07b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37243 } } }
I20260927 10:57:35.004195 3465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ed56951cbe4419781dfd0c9ceede07b [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:35.004426 3465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ed56951cbe4419781dfd0c9ceede07b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ed56951cbe4419781dfd0c9ceede07b. Latest consensus state: current_term: 1 leader_uuid: "4ed56951cbe4419781dfd0c9ceede07b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed56951cbe4419781dfd0c9ceede07b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37243 } } }
I20260927 10:57:35.004544 3470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:35.004560 3465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ed56951cbe4419781dfd0c9ceede07b [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:35.007020 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:57:35.008098 3470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:35.013554 3470 catalog_manager.cc:1347] Generated new cluster ID: 8fb6f9d3d0b646f396f26107b749a218
I20260927 10:57:35.013617 3470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:35.019073 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:35.010048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:58597 (local address 127.0.0.1:37243)
0927 10:57:35.010176 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:35.010180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:35.010796 (+ 616us) server_negotiation.cc:407] Connection header received
0927 10:57:35.010912 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:35.010915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:35.010961 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:35.011041 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:35.012069 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:35.012557 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:35.013411 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:35.013774 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:35.013878 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:35.015548 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:35.015552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:35.015554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:35.015580 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:35.017274 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:35.017820 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:35.017823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:35.017825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:35.017901 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:35.018234 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:35.018237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:35.018239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:35.018417 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:35.018537 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:35.018840 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 10:57:35.018926 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:57:35.024708 3470 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:35.025679 3470 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:35.031914 3470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ed56951cbe4419781dfd0c9ceede07b: Generated new TSK 0
I20260927 10:57:35.032668 3470 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:57:35.041013 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 3392
W20260927 10:57:35.046303 3377 connection.cc:535] client connection to 127.0.0.1:37243 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37243): Connection reset by peer (error 104)
2026-09-27T10:57:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:35 dist-test-slave-tz51 krb5kdc[3491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:35 dist-test-slave-tz51 krb5kdc[3491](info): set up 2 sockets
Sep 27 10:57:35 dist-test-slave-tz51 krb5kdc[3491](info): commencing operation
krb5kdc: starting...
W20260927 10:57:36.067961 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:57:36 dist-test-slave-tz51 krb5kdc[3491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:57:36.187039 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:36.292847 3504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:36.293079 3504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:36.293118 3504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:36.296903 3504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:36.296948 3504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:36.296967 3504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:36.296981 3504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:36.296995 3504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:36.302488 3504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36775
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:36.303359 3504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:36.304273 3504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:36.312314 3512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:36.312422 3510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:36.312431 3509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:36.312536 3504 server_base.cc:1034] running on GCE node
I20260927 10:57:36.313027 3504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:36.314037 3504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:36.315218 3504 hybrid_clock.cc:648] HybridClock initialized: now 1790506656315207 us; error 25 us; skew 500 ppm
Sep 27 10:57:36 dist-test-slave-tz51 krb5kdc[3491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:36.317797 3504 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:57:36.318578 3504 webserver.cc:460] Webserver started at http://127.0.0.1:38899/ using document root <none> and password file <none>
I20260927 10:57:36.319100 3504 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:36.319195 3504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:36.319404 3504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:36.321311 3504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "b8f7fda80b794003bda6faada27ccfcc"
format_stamp: "Formatted at 2026-09-27 10:57:36 on dist-test-slave-tz51"
server_key: "5d4b06726969e090cb7f25cc30d195b4"
server_key_iv: "c82b8f498b1662b9b366ddfc8b682325"
server_key_version: "encryptionkey@0"
I20260927 10:57:36.321861 3504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "b8f7fda80b794003bda6faada27ccfcc"
format_stamp: "Formatted at 2026-09-27 10:57:36 on dist-test-slave-tz51"
server_key: "5d4b06726969e090cb7f25cc30d195b4"
server_key_iv: "c82b8f498b1662b9b366ddfc8b682325"
server_key_version: "encryptionkey@0"
I20260927 10:57:36.325362 3504 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:57:36.327579 3519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:36.328786 3504 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:57:36.328928 3504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "b8f7fda80b794003bda6faada27ccfcc"
format_stamp: "Formatted at 2026-09-27 10:57:36 on dist-test-slave-tz51"
server_key: "5d4b06726969e090cb7f25cc30d195b4"
server_key_iv: "c82b8f498b1662b9b366ddfc8b682325"
server_key_version: "encryptionkey@0"
I20260927 10:57:36.329031 3504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:36.347529 3504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:36.350309 3504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:36.350508 3504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:36.357292 3504 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36775
I20260927 10:57:36.357316 3571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36775 every 8 connection(s)
I20260927 10:57:36.358199 3504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:36.360735 3572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:36.362541 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 3504
I20260927 10:57:36.362690 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:36.362977 14608 external_mini_cluster.cc:1428] Setting key 77612c584343cabae1550fe61afbbf9e
I20260927 10:57:36.366892 3572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc: Bootstrap starting.
I20260927 10:57:36.369045 3572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:36.369854 3572 log.cc:824] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:36.371531 3575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:36.364352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:33101 (local address 127.0.0.1:36775)
0927 10:57:36.364763 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:36.364777 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:36.364811 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:57:36.365444 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:36.365460 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:36.365777 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:36.366146 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:36.366511 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:36.367481 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:36.368423 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:36.368877 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:36.369042 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:36.369879 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:36.369884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:36.369896 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:36.370308 (+ 412us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:36.370359 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:36.370381 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:36.370513 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:36.370683 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 10:57:36.370815 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":134,"threads_started":1}
I20260927 10:57:36.371774 3572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc: No bootstrap required, opened a new log
I20260927 10:57:36.374295 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:57:36.375761 3572 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f7fda80b794003bda6faada27ccfcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260927 10:57:36.375985 3572 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:36.376066 3572 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f7fda80b794003bda6faada27ccfcc, State: Initialized, Role: FOLLOWER
I20260927 10:57:36.376559 3572 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f7fda80b794003bda6faada27ccfcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260927 10:57:36.376722 3572 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:36.376793 3572 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:36.376889 3572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:36.377718 3572 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f7fda80b794003bda6faada27ccfcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260927 10:57:36.378029 3572 leader_election.cc:302] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8f7fda80b794003bda6faada27ccfcc; no voters:
I20260927 10:57:36.378334 3572 leader_election.cc:288] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:36.378399 3582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:36.378633 3582 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc [term 1 LEADER]: Becoming Leader. State: Replica: b8f7fda80b794003bda6faada27ccfcc, State: Running, Role: LEADER
I20260927 10:57:36.378963 3582 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f7fda80b794003bda6faada27ccfcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260927 10:57:36.379612 3572 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:36.380363 3575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:36.376104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:59551 (local address 127.0.0.1:36775)
0927 10:57:36.376199 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:36.376204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:36.376509 (+ 305us) server_negotiation.cc:407] Connection header received
0927 10:57:36.376613 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:36.376615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:36.376652 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:36.376736 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:36.376984 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:36.377492 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:36.378371 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:36.378681 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:36.378778 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:36.379875 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:36.379878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:36.379881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:36.379910 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:36.379934 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:36.379945 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:36.380017 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:36.380198 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 10:57:36.380231 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 10:57:36.380494 3582 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8f7fda80b794003bda6faada27ccfcc. Latest consensus state: current_term: 1 leader_uuid: "b8f7fda80b794003bda6faada27ccfcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f7fda80b794003bda6faada27ccfcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } } }
I20260927 10:57:36.381237 3582 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:36.381102 3583 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8f7fda80b794003bda6faada27ccfcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f7fda80b794003bda6faada27ccfcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } } }
I20260927 10:57:36.381537 3583 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc [sys.catalog]: This master's current role is: LEADER
W20260927 10:57:36.384464 3597 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:57:36.384555 3597 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:57:36.384630 3595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:36.385283 3595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:36.390384 3595 catalog_manager.cc:1347] Generated new cluster ID: 555500bf26de4edeae66d686af0d4f9e
I20260927 10:57:36.390440 3595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:36.399178 3595 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:36.400370 3595 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:36.412118 3595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8f7fda80b794003bda6faada27ccfcc: Generated new TSK 0
I20260927 10:57:36.412917 3595 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:57:36.417280 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 3504
W20260927 10:57:36.423318 3489 connection.cc:535] client connection to 127.0.0.1:36775 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36775): Connection reset by peer (error 104)
2026-09-27T10:57:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:36 dist-test-slave-tz51 krb5kdc[3603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:36 dist-test-slave-tz51 krb5kdc[3603](info): set up 2 sockets
Sep 27 10:57:36 dist-test-slave-tz51 krb5kdc[3603](info): commencing operation
krb5kdc: starting...
W20260927 10:57:37.452013 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:57:37 dist-test-slave-tz51 krb5kdc[3603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:57:37.571301 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:37.679459 3616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:37.679788 3616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:37.679891 3616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:37.684494 3616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:37.684578 3616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:37.684618 3616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:37.684656 3616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:37.684692 3616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:37.689870 3616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32831
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:37.690897 3616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:37.691995 3616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:37.700278 3622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:37.700285 3624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:37.700338 3616 server_base.cc:1034] running on GCE node
W20260927 10:57:37.700285 3621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:37.701179 3616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:37.702281 3616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:37.703466 3616 hybrid_clock.cc:648] HybridClock initialized: now 1790506657703442 us; error 36 us; skew 500 ppm
Sep 27 10:57:37 dist-test-slave-tz51 krb5kdc[3603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:37.710682 3616 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:57:37.711603 3616 webserver.cc:460] Webserver started at http://127.0.0.1:33947/ using document root <none> and password file <none>
I20260927 10:57:37.712182 3616 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:37.712244 3616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:37.712404 3616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:37.714340 3616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "6fd0ac58c5644599abe1bbeef67db8a3"
format_stamp: "Formatted at 2026-09-27 10:57:37 on dist-test-slave-tz51"
server_key: "4b62e5447768fd23968d6617ecfaa612"
server_key_iv: "fea4e388584f558a3e2e8cdc2fe6be49"
server_key_version: "encryptionkey@0"
I20260927 10:57:37.714836 3616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "6fd0ac58c5644599abe1bbeef67db8a3"
format_stamp: "Formatted at 2026-09-27 10:57:37 on dist-test-slave-tz51"
server_key: "4b62e5447768fd23968d6617ecfaa612"
server_key_iv: "fea4e388584f558a3e2e8cdc2fe6be49"
server_key_version: "encryptionkey@0"
I20260927 10:57:37.718436 3616 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:57:37.720829 3631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:37.722069 3616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:57:37.722216 3616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "6fd0ac58c5644599abe1bbeef67db8a3"
format_stamp: "Formatted at 2026-09-27 10:57:37 on dist-test-slave-tz51"
server_key: "4b62e5447768fd23968d6617ecfaa612"
server_key_iv: "fea4e388584f558a3e2e8cdc2fe6be49"
server_key_version: "encryptionkey@0"
I20260927 10:57:37.722326 3616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:37.726816 3616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:37.729794 3616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:37.729978 3616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:37.736896 3616 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32831
I20260927 10:57:37.736940 3683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32831 every 8 connection(s)
I20260927 10:57:37.737826 3616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:37.740540 3684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:37.745761 3684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3: Bootstrap starting.
I20260927 10:57:37.747371 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 3616
I20260927 10:57:37.747507 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:37.747828 14608 external_mini_cluster.cc:1428] Setting key 6148cf6e5d42d709bca74c3dc6d08c38
I20260927 10:57:37.748602 3684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:37.749523 3684 log.cc:824] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:37.751842 3684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3: No bootstrap required, opened a new log
I20260927 10:57:37.753695 3687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:37.749070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:38313 (local address 127.0.0.1:32831)
0927 10:57:37.749737 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:37.749753 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:37.749795 (+ 42us) server_negotiation.cc:407] Connection header received
0927 10:57:37.750723 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:37.750745 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:37.751095 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:37.751450 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:37.752268 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:37.752276 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:37.752292 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:37.752304 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:37.752364 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:37.752389 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:37.752502 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:37.752622 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 10:57:37.752783 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":125,"threads_started":1}
I20260927 10:57:37.756392 3684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd0ac58c5644599abe1bbeef67db8a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32831 } }
I20260927 10:57:37.756604 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:57:37.756675 3684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:37.756745 3684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fd0ac58c5644599abe1bbeef67db8a3, State: Initialized, Role: FOLLOWER
I20260927 10:57:37.757176 3684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd0ac58c5644599abe1bbeef67db8a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32831 } }
I20260927 10:57:37.757294 3684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:37.757373 3684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:37.757483 3684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:37.758484 3684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd0ac58c5644599abe1bbeef67db8a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32831 } }
I20260927 10:57:37.758870 3684 leader_election.cc:302] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fd0ac58c5644599abe1bbeef67db8a3; no voters:
I20260927 10:57:37.759209 3684 leader_election.cc:288] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:37.759388 3694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:37.759600 3694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3 [term 1 LEADER]: Becoming Leader. State: Replica: 6fd0ac58c5644599abe1bbeef67db8a3, State: Running, Role: LEADER
I20260927 10:57:37.760042 3694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd0ac58c5644599abe1bbeef67db8a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32831 } }
I20260927 10:57:37.760586 3687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:37.758734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:46827 (local address 127.0.0.1:32831)
0927 10:57:37.758882 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:37.758886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:37.759035 (+ 149us) server_negotiation.cc:407] Connection header received
0927 10:57:37.759183 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:37.759186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:37.759223 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:37.759309 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:37.760196 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:37.760199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:37.760202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:37.760216 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:37.760250 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:37.760263 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:37.760351 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:37.760444 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 10:57:37.760482 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 10:57:37.760607 3684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:37.761845 3695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fd0ac58c5644599abe1bbeef67db8a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd0ac58c5644599abe1bbeef67db8a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32831 } } }
I20260927 10:57:37.761976 3695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:37.762287 3696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fd0ac58c5644599abe1bbeef67db8a3. Latest consensus state: current_term: 1 leader_uuid: "6fd0ac58c5644599abe1bbeef67db8a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd0ac58c5644599abe1bbeef67db8a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32831 } } }
I20260927 10:57:37.762373 3696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3 [sys.catalog]: This master's current role is: LEADER
W20260927 10:57:37.764983 3709 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:57:37.765079 3709 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:57:37.765147 3705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:37.765800 3705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:37.771168 3705 catalog_manager.cc:1347] Generated new cluster ID: d93629cad87b47378c70693dca6d7231
I20260927 10:57:37.771232 3705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:37.781885 3705 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:37.782191 3705 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:37.794282 3705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fd0ac58c5644599abe1bbeef67db8a3: Generated new TSK 0
I20260927 10:57:37.794929 3705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:57:37.805058 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 3616
2026-09-27T10:57:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:37 dist-test-slave-tz51 krb5kdc[3715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:37 dist-test-slave-tz51 krb5kdc[3715](info): set up 2 sockets
Sep 27 10:57:37 dist-test-slave-tz51 krb5kdc[3715](info): commencing operation
krb5kdc: starting...
W20260927 10:57:38.844002 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:57:38 dist-test-slave-tz51 krb5kdc[3715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:57:38.962782 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:37871
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:45231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:37871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:39.070477 3728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:39.070777 3728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:39.070850 3728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:39.074851 3728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:39.074936 3728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:39.074975 3728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:39.075037 3728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:39.075078 3728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:39.080192 3728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:45231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:37871
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:37871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:39.081152 3728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:39.082163 3728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:39.090420 3733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:39.090528 3728 server_base.cc:1034] running on GCE node
W20260927 10:57:39.090431 3736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:39.090420 3734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:39.091275 3728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:39.092350 3728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:39.093538 3728 hybrid_clock.cc:648] HybridClock initialized: now 1790506659093524 us; error 42 us; skew 500 ppm
Sep 27 10:57:39 dist-test-slave-tz51 krb5kdc[3715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506659, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:39.096247 3728 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:57:39.097101 3728 webserver.cc:460] Webserver started at http://127.14.68.62:41165/ using document root <none> and password file <none>
I20260927 10:57:39.097585 3728 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:39.097651 3728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:39.097805 3728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:39.099738 3728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "e2dba4553f434b62a08e55a6af1358f4"
format_stamp: "Formatted at 2026-09-27 10:57:39 on dist-test-slave-tz51"
server_key: "f8bda9be0cc52e7827b2f1c61e85ad42"
server_key_iv: "0b8e7334afa44b9a036189ce41bdd413"
server_key_version: "encryptionkey@0"
I20260927 10:57:39.100282 3728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "e2dba4553f434b62a08e55a6af1358f4"
format_stamp: "Formatted at 2026-09-27 10:57:39 on dist-test-slave-tz51"
server_key: "f8bda9be0cc52e7827b2f1c61e85ad42"
server_key_iv: "0b8e7334afa44b9a036189ce41bdd413"
server_key_version: "encryptionkey@0"
I20260927 10:57:39.103803 3728 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:57:39.106138 3743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:39.107302 3728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:39.107434 3728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "e2dba4553f434b62a08e55a6af1358f4"
format_stamp: "Formatted at 2026-09-27 10:57:39 on dist-test-slave-tz51"
server_key: "f8bda9be0cc52e7827b2f1c61e85ad42"
server_key_iv: "0b8e7334afa44b9a036189ce41bdd413"
server_key_version: "encryptionkey@0"
I20260927 10:57:39.107534 3728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:39.118156 3728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:39.121392 3728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:39.121587 3728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:39.128597 3728 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:37871
I20260927 10:57:39.128633 3795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:37871 every 8 connection(s)
I20260927 10:57:39.129499 3728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:39.132282 3796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:39.137617 3796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2dba4553f434b62a08e55a6af1358f4: Bootstrap starting.
I20260927 10:57:39.138717 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 3728
I20260927 10:57:39.138871 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:39.139199 14608 external_mini_cluster.cc:1428] Setting key d297839426ef04520d98dbec34af8768
I20260927 10:57:39.140523 3796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2dba4553f434b62a08e55a6af1358f4: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:39.141232 3796 log.cc:824] T 00000000000000000000000000000000 P e2dba4553f434b62a08e55a6af1358f4: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:39.143059 3796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2dba4553f434b62a08e55a6af1358f4: No bootstrap required, opened a new log
I20260927 10:57:39.146358 3796 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2dba4553f434b62a08e55a6af1358f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2dba4553f434b62a08e55a6af1358f4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 37871 } }
I20260927 10:57:39.146600 3796 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2dba4553f434b62a08e55a6af1358f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:39.146687 3796 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2dba4553f434b62a08e55a6af1358f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dba4553f434b62a08e55a6af1358f4, State: Initialized, Role: FOLLOWER
I20260927 10:57:39.147099 3796 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2dba4553f434b62a08e55a6af1358f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2dba4553f434b62a08e55a6af1358f4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 37871 } }
I20260927 10:57:39.147253 3796 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2dba4553f434b62a08e55a6af1358f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:39.147337 3796 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2dba4553f434b62a08e55a6af1358f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:39.147441 3796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2dba4553f434b62a08e55a6af1358f4 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 10:57:39 dist-test-slave-tz51 krb5kdc[3715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:57:39.148602 3796 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2dba4553f434b62a08e55a6af1358f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2dba4553f434b62a08e55a6af1358f4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 37871 } }
I20260927 10:57:39.148943 3796 leader_election.cc:302] T 00000000000000000000000000000000 P e2dba4553f434b62a08e55a6af1358f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2dba4553f434b62a08e55a6af1358f4; no voters:
I20260927 10:57:39.149192 3796 leader_election.cc:288] T 00000000000000000000000000000000 P e2dba4553f434b62a08e55a6af1358f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:39.149299 3801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2dba4553f434b62a08e55a6af1358f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:39.149536 3801 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2dba4553f434b62a08e55a6af1358f4 [term 1 LEADER]: Becoming Leader. State: Replica: e2dba4553f434b62a08e55a6af1358f4, State: Running, Role: LEADER
I20260927 10:57:39.149893 3801 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2dba4553f434b62a08e55a6af1358f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2dba4553f434b62a08e55a6af1358f4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 37871 } }
I20260927 10:57:39.150532 3796 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2dba4553f434b62a08e55a6af1358f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:39.151544 3803 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2dba4553f434b62a08e55a6af1358f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2dba4553f434b62a08e55a6af1358f4. Latest consensus state: current_term: 1 leader_uuid: "e2dba4553f434b62a08e55a6af1358f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2dba4553f434b62a08e55a6af1358f4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 37871 } } }
I20260927 10:57:39.151672 3803 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2dba4553f434b62a08e55a6af1358f4 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:39.151856 3802 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2dba4553f434b62a08e55a6af1358f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2dba4553f434b62a08e55a6af1358f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2dba4553f434b62a08e55a6af1358f4" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 37871 } } }
I20260927 10:57:39.151942 3802 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2dba4553f434b62a08e55a6af1358f4 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:39.152191 3810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:39.154559 3799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:39.140300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49507 (local address 127.14.68.62:37871)
0927 10:57:39.140788 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:39.140811 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:39.140846 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:57:39.141519 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:39.141537 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:39.141977 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:39.142317 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:39.143117 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:39.144206 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:39.145075 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:39.145534 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:39.145717 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:39.148776 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:39.148783 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:39.148804 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:39.148830 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:39.150833 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:39.151582 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:39.151588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:39.151595 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:39.151763 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:39.152220 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:39.152225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:39.152229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:39.152786 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:39.153063 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:39.153433 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 10:57:39.153663 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":140,"threads_started":1}
I20260927 10:57:39.155301 3810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:39.158095 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:57:39.161193 3810 catalog_manager.cc:1347] Generated new cluster ID: d9689bea42954fcb828070b25e22dea1
I20260927 10:57:39.161257 3810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:39.168677 3799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:39.161049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42073 (local address 127.14.68.62:37871)
0927 10:57:39.161158 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:39.161163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:39.161176 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:57:39.161214 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:39.161216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:39.161259 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:39.161339 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:39.162059 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:39.162750 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:39.163542 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:39.163870 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:39.163982 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:39.165373 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:39.165377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:39.165380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:39.165404 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:39.166872 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:39.167347 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:39.167351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:39.167353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:39.167432 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:39.167810 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:39.167814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:39.167816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:39.167983 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:39.168148 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:39.168407 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 10:57:39.168499 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:57:39.177194 3810 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:39.178061 3810 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:39.200416 3810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2dba4553f434b62a08e55a6af1358f4: Generated new TSK 0
I20260927 10:57:39.201205 3810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:57:39.205389 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 3728
W20260927 10:57:39.210122 3713 connection.cc:535] client connection to 127.14.68.62:37871 recv error: Network error: failed to read from TLS socket (remote: 127.14.68.62:37871): Connection reset by peer (error 104)
2026-09-27T10:57:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:39 dist-test-slave-tz51 krb5kdc[3827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:39 dist-test-slave-tz51 krb5kdc[3827](info): set up 2 sockets
Sep 27 10:57:39 dist-test-slave-tz51 krb5kdc[3827](info): commencing operation
krb5kdc: starting...
W20260927 10:57:40.251947 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:57:40 dist-test-slave-tz51 krb5kdc[3827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:57:40.370851 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:38879
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:35143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:38879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:40.478336 3840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:40.478592 3840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:40.478633 3840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:40.482669 3840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:40.482738 3840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:40.482757 3840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:40.482774 3840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:40.482789 3840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:40.488286 3840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:35143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:38879
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:38879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:40.489339 3840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:40.490345 3840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:57:40.498546 3840 server_base.cc:1034] running on GCE node
W20260927 10:57:40.498523 3845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:40.498529 3846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:40.498795 3848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:40.499370 3840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:40.500447 3840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:40.501627 3840 hybrid_clock.cc:648] HybridClock initialized: now 1790506660501592 us; error 47 us; skew 500 ppm
Sep 27 10:57:40 dist-test-slave-tz51 krb5kdc[3827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506660, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:40.504315 3840 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:57:40.505138 3840 webserver.cc:460] Webserver started at http://127.14.68.62:35781/ using document root <none> and password file <none>
I20260927 10:57:40.505708 3840 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:40.505797 3840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:40.506033 3840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:40.508037 3840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "cdc2265677e64fb995d6afcfe7d714ff"
format_stamp: "Formatted at 2026-09-27 10:57:40 on dist-test-slave-tz51"
server_key: "9be3b9c35c51ef06039f9c67e3e7ac8c"
server_key_iv: "2fc55abfde5db45a12f6cdc0e0152d3b"
server_key_version: "encryptionkey@0"
I20260927 10:57:40.508590 3840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "cdc2265677e64fb995d6afcfe7d714ff"
format_stamp: "Formatted at 2026-09-27 10:57:40 on dist-test-slave-tz51"
server_key: "9be3b9c35c51ef06039f9c67e3e7ac8c"
server_key_iv: "2fc55abfde5db45a12f6cdc0e0152d3b"
server_key_version: "encryptionkey@0"
I20260927 10:57:40.512127 3840 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:57:40.514328 3855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:40.515511 3840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:40.515646 3840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "cdc2265677e64fb995d6afcfe7d714ff"
format_stamp: "Formatted at 2026-09-27 10:57:40 on dist-test-slave-tz51"
server_key: "9be3b9c35c51ef06039f9c67e3e7ac8c"
server_key_iv: "2fc55abfde5db45a12f6cdc0e0152d3b"
server_key_version: "encryptionkey@0"
I20260927 10:57:40.515815 3840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:40.530753 3840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:40.533907 3840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:40.534111 3840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:40.540845 3840 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:38879
I20260927 10:57:40.540905 3907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:38879 every 8 connection(s)
I20260927 10:57:40.541786 3840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:40.544420 3908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:40.546240 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 3840
I20260927 10:57:40.546386 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:40.546664 14608 external_mini_cluster.cc:1428] Setting key b1c993e9767bc52c29b5b64dc9cd86a6
I20260927 10:57:40.550763 3908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff: Bootstrap starting.
I20260927 10:57:40.553424 3908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:40.554198 3908 log.cc:824] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:40.555475 3911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:40.548045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36851 (local address 127.14.68.62:38879)
0927 10:57:40.548452 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:40.548467 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:40.548503 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:57:40.549136 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:40.549152 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:40.549472 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:40.549742 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:40.550102 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:40.551080 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:40.552302 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:40.552757 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:40.552874 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:40.553761 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:40.553766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:40.553778 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:40.554177 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:40.554229 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:40.554250 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:40.554433 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:40.554612 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 10:57:40.554753 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":141,"threads_started":1}
I20260927 10:57:40.556259 3908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff: No bootstrap required, opened a new log
I20260927 10:57:40.558593 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:57:40.560961 3908 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc2265677e64fb995d6afcfe7d714ff" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38879 } }
I20260927 10:57:40.561259 3908 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:40.561323 3908 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc2265677e64fb995d6afcfe7d714ff, State: Initialized, Role: FOLLOWER
I20260927 10:57:40.561755 3908 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc2265677e64fb995d6afcfe7d714ff" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38879 } }
I20260927 10:57:40.561872 3908 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:40.561956 3908 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:40.562067 3908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:40.562966 3908 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc2265677e64fb995d6afcfe7d714ff" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38879 } }
I20260927 10:57:40.563316 3908 leader_election.cc:302] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdc2265677e64fb995d6afcfe7d714ff; no voters:
I20260927 10:57:40.563611 3908 leader_election.cc:288] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:40.563748 3918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:40.563977 3918 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff [term 1 LEADER]: Becoming Leader. State: Replica: cdc2265677e64fb995d6afcfe7d714ff, State: Running, Role: LEADER
I20260927 10:57:40.564361 3918 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc2265677e64fb995d6afcfe7d714ff" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38879 } }
I20260927 10:57:40.564942 3908 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:40.565754 3911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:40.560875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45923 (local address 127.14.68.62:38879)
0927 10:57:40.561010 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:40.561013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:40.561145 (+ 132us) server_negotiation.cc:407] Connection header received
0927 10:57:40.561280 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:40.561283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:40.561351 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:40.561449 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:40.561716 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:40.562411 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:40.563314 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:40.563670 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:40.563799 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:40.564953 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:40.564960 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:40.564965 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:40.564978 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:40.565004 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:40.565018 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:40.565106 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:40.565542 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 10:57:40.565576 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:57:40.566358 3919 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdc2265677e64fb995d6afcfe7d714ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc2265677e64fb995d6afcfe7d714ff" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38879 } } }
I20260927 10:57:40.566483 3919 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:40.566829 3920 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdc2265677e64fb995d6afcfe7d714ff. Latest consensus state: current_term: 1 leader_uuid: "cdc2265677e64fb995d6afcfe7d714ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc2265677e64fb995d6afcfe7d714ff" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38879 } } }
I20260927 10:57:40.567039 3920 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff [sys.catalog]: This master's current role is: LEADER
W20260927 10:57:40.569618 3933 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:57:40.569680 3933 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:57:40.569759 3930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:40.570432 3930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:40.575937 3930 catalog_manager.cc:1347] Generated new cluster ID: 69c7472b812c47f7ab502172b70a6153
I20260927 10:57:40.576004 3930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:40.586021 3930 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:40.586897 3930 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:40.591434 3930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdc2265677e64fb995d6afcfe7d714ff: Generated new TSK 0
I20260927 10:57:40.592114 3930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:57:40.595724 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 3840
W20260927 10:57:40.600920 3825 connection.cc:535] client connection to 127.14.68.62:38879 recv error: Network error: failed to read from TLS socket (remote: 127.14.68.62:38879): Connection reset by peer (error 104)
2026-09-27T10:57:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:40 dist-test-slave-tz51 krb5kdc[3939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:40 dist-test-slave-tz51 krb5kdc[3939](info): set up 2 sockets
Sep 27 10:57:40 dist-test-slave-tz51 krb5kdc[3939](info): commencing operation
krb5kdc: starting...
W20260927 10:57:41.632069 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:57:41 dist-test-slave-tz51 krb5kdc[3939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:57:41.755620 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:38589
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:43421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:38589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:41.864470 3952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:41.864782 3952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:41.864880 3952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:41.868897 3952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:41.868947 3952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:41.868966 3952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:41.868983 3952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:41.868999 3952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:41.874246 3952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:43421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:38589
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:38589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:41.875273 3952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:41.876333 3952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:41.884477 3958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:41.884511 3960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:41.884583 3952 server_base.cc:1034] running on GCE node
W20260927 10:57:41.884477 3957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:41.885226 3952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:41.886318 3952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:41.887511 3952 hybrid_clock.cc:648] HybridClock initialized: now 1790506661887492 us; error 32 us; skew 500 ppm
Sep 27 10:57:41 dist-test-slave-tz51 krb5kdc[3939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506661, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:41.890280 3952 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:57:41.891152 3952 webserver.cc:460] Webserver started at http://127.14.68.62:41627/ using document root <none> and password file <none>
I20260927 10:57:41.891752 3952 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:41.891839 3952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:41.892066 3952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:41.894016 3952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "079505707cc746f2a94e9727616fe21c"
format_stamp: "Formatted at 2026-09-27 10:57:41 on dist-test-slave-tz51"
server_key: "3cc9011341eefd5c6cc7f8801d8d8e4a"
server_key_iv: "fbb95570a694e4ee88a7d762c2df670a"
server_key_version: "encryptionkey@0"
I20260927 10:57:41.894568 3952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "079505707cc746f2a94e9727616fe21c"
format_stamp: "Formatted at 2026-09-27 10:57:41 on dist-test-slave-tz51"
server_key: "3cc9011341eefd5c6cc7f8801d8d8e4a"
server_key_iv: "fbb95570a694e4ee88a7d762c2df670a"
server_key_version: "encryptionkey@0"
I20260927 10:57:41.898114 3952 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:57:41.900418 3967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:41.901597 3952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:41.901746 3952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "079505707cc746f2a94e9727616fe21c"
format_stamp: "Formatted at 2026-09-27 10:57:41 on dist-test-slave-tz51"
server_key: "3cc9011341eefd5c6cc7f8801d8d8e4a"
server_key_iv: "fbb95570a694e4ee88a7d762c2df670a"
server_key_version: "encryptionkey@0"
I20260927 10:57:41.901852 3952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:41.906491 3952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:41.909647 3952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:41.909840 3952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:41.916424 3952 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:38589
I20260927 10:57:41.916507 4019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:38589 every 8 connection(s)
I20260927 10:57:41.917325 3952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:41.920195 4020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:41.921705 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 3952
I20260927 10:57:41.921842 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:41.922147 14608 external_mini_cluster.cc:1428] Setting key 16e32b396bc4d77646edd2aa37a7a460
I20260927 10:57:41.926412 4020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 079505707cc746f2a94e9727616fe21c: Bootstrap starting.
I20260927 10:57:41.927232 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:41.923483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42401 (local address 127.14.68.62:38589)
0927 10:57:41.923940 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:41.923954 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:41.923989 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:57:41.924597 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:41.924612 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:41.924891 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:41.925217 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:41.926024 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:41.926029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:41.926041 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:41.926051 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:41.926105 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:41.926123 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:41.926276 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:41.926349 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 10:57:41.926528 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":147,"threads_started":1}
I20260927 10:57:41.928934 4020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 079505707cc746f2a94e9727616fe21c: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:41.929849 4020 log.cc:824] T 00000000000000000000000000000000 P 079505707cc746f2a94e9727616fe21c: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:41.930244 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:57:41.931926 4020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 079505707cc746f2a94e9727616fe21c: No bootstrap required, opened a new log
I20260927 10:57:41.933848 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:41.932297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39119 (local address 127.14.68.62:38589)
0927 10:57:41.932417 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:41.932422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:41.932588 (+ 166us) server_negotiation.cc:407] Connection header received
0927 10:57:41.932673 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:41.932676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:41.932716 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:41.932801 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:41.933487 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:41.933491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:41.933494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:41.933506 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:41.933541 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:41.933554 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:41.933642 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:41.933696 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 10:57:41.933733 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:57:41.936484 4020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 079505707cc746f2a94e9727616fe21c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079505707cc746f2a94e9727616fe21c" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38589 } }
I20260927 10:57:41.936702 4020 raft_consensus.cc:374] T 00000000000000000000000000000000 P 079505707cc746f2a94e9727616fe21c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:41.936774 4020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 079505707cc746f2a94e9727616fe21c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 079505707cc746f2a94e9727616fe21c, State: Initialized, Role: FOLLOWER
I20260927 10:57:41.937206 4020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 079505707cc746f2a94e9727616fe21c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079505707cc746f2a94e9727616fe21c" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38589 } }
I20260927 10:57:41.937346 4020 raft_consensus.cc:388] T 00000000000000000000000000000000 P 079505707cc746f2a94e9727616fe21c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:41.937409 4020 raft_consensus.cc:482] T 00000000000000000000000000000000 P 079505707cc746f2a94e9727616fe21c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:41.937522 4020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 079505707cc746f2a94e9727616fe21c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:41.938472 4020 raft_consensus.cc:504] T 00000000000000000000000000000000 P 079505707cc746f2a94e9727616fe21c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079505707cc746f2a94e9727616fe21c" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38589 } }
I20260927 10:57:41.938797 4020 leader_election.cc:302] T 00000000000000000000000000000000 P 079505707cc746f2a94e9727616fe21c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 079505707cc746f2a94e9727616fe21c; no voters:
I20260927 10:57:41.939132 4020 leader_election.cc:288] T 00000000000000000000000000000000 P 079505707cc746f2a94e9727616fe21c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:41.939234 4030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 079505707cc746f2a94e9727616fe21c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:41.939466 4030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 079505707cc746f2a94e9727616fe21c [term 1 LEADER]: Becoming Leader. State: Replica: 079505707cc746f2a94e9727616fe21c, State: Running, Role: LEADER
I20260927 10:57:41.939836 4030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 079505707cc746f2a94e9727616fe21c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079505707cc746f2a94e9727616fe21c" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38589 } }
I20260927 10:57:41.940399 4020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 079505707cc746f2a94e9727616fe21c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:41.941723 4031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 079505707cc746f2a94e9727616fe21c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "079505707cc746f2a94e9727616fe21c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079505707cc746f2a94e9727616fe21c" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38589 } } }
I20260927 10:57:41.941725 4032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 079505707cc746f2a94e9727616fe21c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 079505707cc746f2a94e9727616fe21c. Latest consensus state: current_term: 1 leader_uuid: "079505707cc746f2a94e9727616fe21c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079505707cc746f2a94e9727616fe21c" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 38589 } } }
I20260927 10:57:41.941874 4032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 079505707cc746f2a94e9727616fe21c [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:41.941874 4031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 079505707cc746f2a94e9727616fe21c [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:41.942631 4039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:41.945302 4039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:41.950367 4039 catalog_manager.cc:1347] Generated new cluster ID: f20aee07a2164787958e813f3ad303c0
I20260927 10:57:41.950443 4039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:41.957341 4039 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:41.957630 4039 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:41.962767 4039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 079505707cc746f2a94e9727616fe21c: Generated new TSK 0
I20260927 10:57:41.963378 4039 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:57:41.968501 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 3952
2026-09-27T10:57:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1372 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:42 dist-test-slave-tz51 krb5kdc[4051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:42 dist-test-slave-tz51 krb5kdc[4051](info): set up 2 sockets
Sep 27 10:57:42 dist-test-slave-tz51 krb5kdc[4051](info): commencing operation
krb5kdc: starting...
W20260927 10:57:43.003963 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:57:43 dist-test-slave-tz51 krb5kdc[4051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:57:43.122808 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:36353
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:36353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:43.229593 4064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:43.229887 4064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:43.229954 4064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:43.233855 4064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:43.233939 4064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:43.233978 4064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:43.234046 4064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:43.234089 4064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:43.239065 4064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:36353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:36353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:43.240069 4064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:43.241051 4064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:43.249393 4069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:43.249451 4072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:43.249434 4070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:43.249571 4064 server_base.cc:1034] running on GCE node
I20260927 10:57:43.250174 4064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:43.251281 4064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:43.252457 4064 hybrid_clock.cc:648] HybridClock initialized: now 1790506663252440 us; error 30 us; skew 500 ppm
Sep 27 10:57:43 dist-test-slave-tz51 krb5kdc[4051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506663, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:43.260799 4064 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:57:43.261749 4064 webserver.cc:460] Webserver started at http://127.14.68.62:46419/ using document root <none> and password file <none>
I20260927 10:57:43.262329 4064 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:43.262423 4064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:43.262670 4064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:43.264851 4064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "9f1f520d0124467cad66236d067d9215"
format_stamp: "Formatted at 2026-09-27 10:57:43 on dist-test-slave-tz51"
server_key: "bf256684f8d194929731c8f58b49fd5a"
server_key_iv: "7d6bf913257b2288445b5a68d3e2cb0e"
server_key_version: "encryptionkey@0"
I20260927 10:57:43.265443 4064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "9f1f520d0124467cad66236d067d9215"
format_stamp: "Formatted at 2026-09-27 10:57:43 on dist-test-slave-tz51"
server_key: "bf256684f8d194929731c8f58b49fd5a"
server_key_iv: "7d6bf913257b2288445b5a68d3e2cb0e"
server_key_version: "encryptionkey@0"
I20260927 10:57:43.269443 4064 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 10:57:43.271972 4079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:43.273237 4064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:57:43.273394 4064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "9f1f520d0124467cad66236d067d9215"
format_stamp: "Formatted at 2026-09-27 10:57:43 on dist-test-slave-tz51"
server_key: "bf256684f8d194929731c8f58b49fd5a"
server_key_iv: "7d6bf913257b2288445b5a68d3e2cb0e"
server_key_version: "encryptionkey@0"
I20260927 10:57:43.273512 4064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:43.294049 4064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:43.296914 4064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:43.297127 4064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:43.303927 4064 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:36353
I20260927 10:57:43.303969 4131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:36353 every 8 connection(s)
I20260927 10:57:43.304905 4064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:43.307585 4132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:43.308557 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 4064
I20260927 10:57:43.308717 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:43.309007 14608 external_mini_cluster.cc:1428] Setting key 950f4caed2fbbeb8bd1be2dfa163d770
I20260927 10:57:43.314268 4132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f1f520d0124467cad66236d067d9215: Bootstrap starting.
I20260927 10:57:43.316732 4132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f1f520d0124467cad66236d067d9215: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:43.317698 4132 log.cc:824] T 00000000000000000000000000000000 P 9f1f520d0124467cad66236d067d9215: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:57:43 dist-test-slave-tz51 krb5kdc[4051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:57:43.319887 4132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f1f520d0124467cad66236d067d9215: No bootstrap required, opened a new log
I20260927 10:57:43.324397 4135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:43.310395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46325 (local address 127.14.68.62:36353)
0927 10:57:43.310820 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:43.310843 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:43.310888 (+ 45us) server_negotiation.cc:407] Connection header received
0927 10:57:43.311757 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:43.311781 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:43.312201 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:43.312565 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:43.313272 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:43.314687 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:43.315509 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:43.316194 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:43.316353 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:43.319281 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:43.319287 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:43.319302 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:43.319333 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:43.321327 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:43.321747 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:43.321753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:43.321759 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:43.321896 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:43.322173 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:43.322177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:43.322180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:43.322741 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:43.322927 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:43.323122 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 10:57:43.323412 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":110,"threads_started":1}
I20260927 10:57:43.324481 4132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f1f520d0124467cad66236d067d9215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1f520d0124467cad66236d067d9215" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 36353 } }
I20260927 10:57:43.324735 4132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f1f520d0124467cad66236d067d9215 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:43.324797 4132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f1f520d0124467cad66236d067d9215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f1f520d0124467cad66236d067d9215, State: Initialized, Role: FOLLOWER
I20260927 10:57:43.325229 4132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f1f520d0124467cad66236d067d9215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1f520d0124467cad66236d067d9215" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 36353 } }
I20260927 10:57:43.325383 4132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f1f520d0124467cad66236d067d9215 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:43.325451 4132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f1f520d0124467cad66236d067d9215 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:43.325575 4132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f1f520d0124467cad66236d067d9215 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:43.326610 4132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f1f520d0124467cad66236d067d9215 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1f520d0124467cad66236d067d9215" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 36353 } }
I20260927 10:57:43.326962 4132 leader_election.cc:302] T 00000000000000000000000000000000 P 9f1f520d0124467cad66236d067d9215 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f1f520d0124467cad66236d067d9215; no voters:
I20260927 10:57:43.327273 4132 leader_election.cc:288] T 00000000000000000000000000000000 P 9f1f520d0124467cad66236d067d9215 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:43.327327 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:57:43.327365 4137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f1f520d0124467cad66236d067d9215 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:43.327581 4137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f1f520d0124467cad66236d067d9215 [term 1 LEADER]: Becoming Leader. State: Replica: 9f1f520d0124467cad66236d067d9215, State: Running, Role: LEADER
I20260927 10:57:43.327898 4137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f1f520d0124467cad66236d067d9215 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1f520d0124467cad66236d067d9215" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 36353 } }
I20260927 10:57:43.328408 4132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f1f520d0124467cad66236d067d9215 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:43.329514 4139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f1f520d0124467cad66236d067d9215 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f1f520d0124467cad66236d067d9215. Latest consensus state: current_term: 1 leader_uuid: "9f1f520d0124467cad66236d067d9215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1f520d0124467cad66236d067d9215" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 36353 } } }
I20260927 10:57:43.329638 4139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f1f520d0124467cad66236d067d9215 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:43.329917 4138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f1f520d0124467cad66236d067d9215 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f1f520d0124467cad66236d067d9215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1f520d0124467cad66236d067d9215" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 36353 } } }
I20260927 10:57:43.330015 4138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f1f520d0124467cad66236d067d9215 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:43.330024 4150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:43.333211 4150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:43.339047 4150 catalog_manager.cc:1347] Generated new cluster ID: 717efdabad6a4c3e8e91cba6600ebfc6
I20260927 10:57:43.339150 4150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:43.339855 4135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:43.330837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41371 (local address 127.14.68.62:36353)
0927 10:57:43.330957 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:43.330961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:43.331215 (+ 254us) server_negotiation.cc:407] Connection header received
0927 10:57:43.331335 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:43.331338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:43.331378 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:43.331457 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:43.332240 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:43.332925 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:43.334054 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:43.334371 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:43.334481 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:43.336096 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:43.336100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:43.336102 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:43.336125 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:43.338024 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:43.338478 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:43.338481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:43.338482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:43.338573 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:43.338880 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:43.338883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:43.338885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:43.339081 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:43.339222 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:43.339560 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 10:57:43.339668 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:57:43.348618 4150 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:43.349778 4150 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:43.357635 4150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f1f520d0124467cad66236d067d9215: Generated new TSK 0
I20260927 10:57:43.358316 4150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:57:43.365154 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 4064
2026-09-27T10:57:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:43 dist-test-slave-tz51 krb5kdc[4163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:43 dist-test-slave-tz51 krb5kdc[4163](info): set up 2 sockets
Sep 27 10:57:43 dist-test-slave-tz51 krb5kdc[4163](info): commencing operation
krb5kdc: starting...
W20260927 10:57:44.404043 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:57:44 dist-test-slave-tz51 krb5kdc[4163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:57:44.527760 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:44403
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:42287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:44403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:44.639256 4176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:44.639503 4176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:44.639544 4176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:44.643628 4176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:44.643677 4176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:44.643710 4176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:44.643728 4176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:44.643741 4176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:44.648895 4176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:42287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:44403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:44403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:44.649904 4176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:44.650943 4176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:44.659667 4182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:44.659672 4184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:44.659646 4181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:44.659734 4176 server_base.cc:1034] running on GCE node
I20260927 10:57:44.660559 4176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:44.661669 4176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:44.662864 4176 hybrid_clock.cc:648] HybridClock initialized: now 1790506664662846 us; error 31 us; skew 500 ppm
Sep 27 10:57:44 dist-test-slave-tz51 krb5kdc[4163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506664, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:44.665704 4176 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:57:44.666638 4176 webserver.cc:460] Webserver started at http://127.14.68.62:34951/ using document root <none> and password file <none>
I20260927 10:57:44.667196 4176 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:44.667289 4176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:44.667514 4176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:44.669615 4176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "0ac8c6a540584bc4a0b5ec7e30f98748"
format_stamp: "Formatted at 2026-09-27 10:57:44 on dist-test-slave-tz51"
server_key: "0308bc29bbc2d3c9240334a2210a30df"
server_key_iv: "124afa91fc793702c1e5f81b1dec812d"
server_key_version: "encryptionkey@0"
I20260927 10:57:44.670181 4176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "0ac8c6a540584bc4a0b5ec7e30f98748"
format_stamp: "Formatted at 2026-09-27 10:57:44 on dist-test-slave-tz51"
server_key: "0308bc29bbc2d3c9240334a2210a30df"
server_key_iv: "124afa91fc793702c1e5f81b1dec812d"
server_key_version: "encryptionkey@0"
I20260927 10:57:44.673797 4176 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:57:44.676133 4191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:44.677246 4176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:57:44.677392 4176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "0ac8c6a540584bc4a0b5ec7e30f98748"
format_stamp: "Formatted at 2026-09-27 10:57:44 on dist-test-slave-tz51"
server_key: "0308bc29bbc2d3c9240334a2210a30df"
server_key_iv: "124afa91fc793702c1e5f81b1dec812d"
server_key_version: "encryptionkey@0"
I20260927 10:57:44.677500 4176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:44.690683 4176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:44.694216 4176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:44.694437 4176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:44.701813 4176 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:44403
I20260927 10:57:44.701839 4243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:44403 every 8 connection(s)
I20260927 10:57:44.702768 4176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:44.703583 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 4176
I20260927 10:57:44.703735 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:44.704017 14608 external_mini_cluster.cc:1428] Setting key 2922960391e8f9e30e291e880b201af5
I20260927 10:57:44.705962 4244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:44.712713 4244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ac8c6a540584bc4a0b5ec7e30f98748: Bootstrap starting.
I20260927 10:57:44.713824 4247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:44.705427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56971 (local address 127.14.68.62:44403)
0927 10:57:44.705854 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:44.705872 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:44.705914 (+ 42us) server_negotiation.cc:407] Connection header received
0927 10:57:44.706783 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:44.706806 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:44.707236 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:44.707595 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:44.708013 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:44.709506 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:44.710425 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:44.711117 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:44.711275 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:44.712076 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:44.712083 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:44.712097 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:44.712590 (+ 493us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:44.712654 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:44.712682 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:44.712786 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:44.712976 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 10:57:44.713173 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":124,"threads_started":1}
I20260927 10:57:44.715157 4244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ac8c6a540584bc4a0b5ec7e30f98748: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:44.716040 4244 log.cc:824] T 00000000000000000000000000000000 P 0ac8c6a540584bc4a0b5ec7e30f98748: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:44.716645 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:57:44.717971 4244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ac8c6a540584bc4a0b5ec7e30f98748: No bootstrap required, opened a new log
I20260927 10:57:44.722256 4244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ac8c6a540584bc4a0b5ec7e30f98748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac8c6a540584bc4a0b5ec7e30f98748" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44403 } }
I20260927 10:57:44.722572 4244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ac8c6a540584bc4a0b5ec7e30f98748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:44.722610 4247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:44.718592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60169 (local address 127.14.68.62:44403)
0927 10:57:44.718715 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:44.718721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:44.718935 (+ 214us) server_negotiation.cc:407] Connection header received
0927 10:57:44.719009 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:44.719012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:44.719053 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:44.719132 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:44.719348 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:44.720211 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:44.721018 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:44.721372 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:44.721475 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:44.722210 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:44.722214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:44.722217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:44.722228 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:44.722257 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:44.722272 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:44.722365 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:44.722426 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 10:57:44.722474 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:57:44.722635 4244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ac8c6a540584bc4a0b5ec7e30f98748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac8c6a540584bc4a0b5ec7e30f98748, State: Initialized, Role: FOLLOWER
I20260927 10:57:44.723085 4244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ac8c6a540584bc4a0b5ec7e30f98748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac8c6a540584bc4a0b5ec7e30f98748" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44403 } }
I20260927 10:57:44.723206 4244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ac8c6a540584bc4a0b5ec7e30f98748 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:44.723291 4244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ac8c6a540584bc4a0b5ec7e30f98748 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:44.723407 4244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ac8c6a540584bc4a0b5ec7e30f98748 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:44.724323 4244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ac8c6a540584bc4a0b5ec7e30f98748 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac8c6a540584bc4a0b5ec7e30f98748" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44403 } }
I20260927 10:57:44.724649 4244 leader_election.cc:302] T 00000000000000000000000000000000 P 0ac8c6a540584bc4a0b5ec7e30f98748 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ac8c6a540584bc4a0b5ec7e30f98748; no voters:
I20260927 10:57:44.724924 4244 leader_election.cc:288] T 00000000000000000000000000000000 P 0ac8c6a540584bc4a0b5ec7e30f98748 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:44.725028 4254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ac8c6a540584bc4a0b5ec7e30f98748 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:44.725282 4254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ac8c6a540584bc4a0b5ec7e30f98748 [term 1 LEADER]: Becoming Leader. State: Replica: 0ac8c6a540584bc4a0b5ec7e30f98748, State: Running, Role: LEADER
I20260927 10:57:44.725651 4254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ac8c6a540584bc4a0b5ec7e30f98748 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac8c6a540584bc4a0b5ec7e30f98748" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44403 } }
I20260927 10:57:44.726224 4244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ac8c6a540584bc4a0b5ec7e30f98748 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:44.727319 4256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac8c6a540584bc4a0b5ec7e30f98748 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ac8c6a540584bc4a0b5ec7e30f98748. Latest consensus state: current_term: 1 leader_uuid: "0ac8c6a540584bc4a0b5ec7e30f98748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac8c6a540584bc4a0b5ec7e30f98748" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44403 } } }
I20260927 10:57:44.727447 4256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac8c6a540584bc4a0b5ec7e30f98748 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:44.727787 4255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac8c6a540584bc4a0b5ec7e30f98748 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ac8c6a540584bc4a0b5ec7e30f98748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac8c6a540584bc4a0b5ec7e30f98748" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 44403 } } }
I20260927 10:57:44.727897 4263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:44.727924 4255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac8c6a540584bc4a0b5ec7e30f98748 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:44.731014 4263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:44.737181 4263 catalog_manager.cc:1347] Generated new cluster ID: 35f1163a84f841b89342bd568475f783
I20260927 10:57:44.737264 4263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:44.744848 4263 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:44.746030 4263 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:44.750581 4263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ac8c6a540584bc4a0b5ec7e30f98748: Generated new TSK 0
I20260927 10:57:44.751267 4263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:57:44.756158 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 4176
2026-09-27T10:57:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1390 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:44 dist-test-slave-tz51 krb5kdc[4275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:44 dist-test-slave-tz51 krb5kdc[4275](info): set up 2 sockets
Sep 27 10:57:44 dist-test-slave-tz51 krb5kdc[4275](info): commencing operation
krb5kdc: starting...
W20260927 10:57:45.792042 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:57:45 dist-test-slave-tz51 krb5kdc[4275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:57:45.914526 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:33521
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:40029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:33521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:46.023595 4288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:46.023912 4288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:46.023993 4288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:46.027963 4288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:46.028046 4288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:46.028105 4288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:46.028143 4288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:46.028182 4288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:46.033231 4288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:40029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:33521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:33521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:46.034199 4288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:46.035204 4288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:46.043427 4296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:46.043452 4293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:46.043733 4288 server_base.cc:1034] running on GCE node
W20260927 10:57:46.043471 4294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:46.044315 4288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:46.045343 4288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:46.046526 4288 hybrid_clock.cc:648] HybridClock initialized: now 1790506666046510 us; error 29 us; skew 500 ppm
Sep 27 10:57:46 dist-test-slave-tz51 krb5kdc[4275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506666, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:46.049187 4288 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:57:46.050071 4288 webserver.cc:460] Webserver started at http://127.14.68.62:42883/ using document root <none> and password file <none>
I20260927 10:57:46.050625 4288 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:46.050716 4288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:46.050945 4288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:46.053062 4288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "c013b182b64d41238e5cd4dc165587a0"
format_stamp: "Formatted at 2026-09-27 10:57:46 on dist-test-slave-tz51"
server_key: "ccca84e292bb610765faa4086239b080"
server_key_iv: "eb6cc28d3cc2ceaa69bf31ece24ec81e"
server_key_version: "encryptionkey@0"
I20260927 10:57:46.053674 4288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "c013b182b64d41238e5cd4dc165587a0"
format_stamp: "Formatted at 2026-09-27 10:57:46 on dist-test-slave-tz51"
server_key: "ccca84e292bb610765faa4086239b080"
server_key_iv: "eb6cc28d3cc2ceaa69bf31ece24ec81e"
server_key_version: "encryptionkey@0"
I20260927 10:57:46.057668 4288 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:57:46.060135 4303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:46.061700 4288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:57:46.061877 4288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "c013b182b64d41238e5cd4dc165587a0"
format_stamp: "Formatted at 2026-09-27 10:57:46 on dist-test-slave-tz51"
server_key: "ccca84e292bb610765faa4086239b080"
server_key_iv: "eb6cc28d3cc2ceaa69bf31ece24ec81e"
server_key_version: "encryptionkey@0"
I20260927 10:57:46.061986 4288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:46.066546 4288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:46.069424 4288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:46.069567 4288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:46.076375 4288 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:33521
I20260927 10:57:46.076393 4355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:33521 every 8 connection(s)
I20260927 10:57:46.077283 4288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:46.080053 4356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:46.080127 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 4288
I20260927 10:57:46.080300 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:46.080596 14608 external_mini_cluster.cc:1428] Setting key e6e0aec8b8914b2d4fd08e2248139aaa
I20260927 10:57:46.085914 4359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:46.082130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43951 (local address 127.14.68.62:33521)
0927 10:57:46.082559 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:46.082574 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:46.082607 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:57:46.083285 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:46.083301 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:46.083568 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:46.083920 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:46.084757 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:46.084762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:46.084774 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:46.084783 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:46.084832 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:46.084850 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:46.084953 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:46.085071 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 10:57:46.085235 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":134,"threads_started":1}
I20260927 10:57:46.086802 4356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c013b182b64d41238e5cd4dc165587a0: Bootstrap starting.
I20260927 10:57:46.088474 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:57:46.089133 4356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c013b182b64d41238e5cd4dc165587a0: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:46.089895 4356 log.cc:824] T 00000000000000000000000000000000 P c013b182b64d41238e5cd4dc165587a0: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:46.091763 4356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c013b182b64d41238e5cd4dc165587a0: No bootstrap required, opened a new log
I20260927 10:57:46.092198 4359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:46.090522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48745 (local address 127.14.68.62:33521)
0927 10:57:46.090664 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:46.090669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:46.090823 (+ 154us) server_negotiation.cc:407] Connection header received
0927 10:57:46.090971 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:46.090974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:46.091010 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:46.091096 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:46.091837 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:46.091841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:46.091843 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:57:46.091856 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:46.091884 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:57:46.091896 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:46.091971 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:46.092042 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 10:57:46.092081 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:57:46.095433 4356 raft_consensus.cc:348] T 00000000000000000000000000000000 P c013b182b64d41238e5cd4dc165587a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c013b182b64d41238e5cd4dc165587a0" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33521 } }
I20260927 10:57:46.095657 4356 raft_consensus.cc:374] T 00000000000000000000000000000000 P c013b182b64d41238e5cd4dc165587a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:46.095754 4356 raft_consensus.cc:729] T 00000000000000000000000000000000 P c013b182b64d41238e5cd4dc165587a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c013b182b64d41238e5cd4dc165587a0, State: Initialized, Role: FOLLOWER
I20260927 10:57:46.096145 4356 consensus_queue.cc:260] T 00000000000000000000000000000000 P c013b182b64d41238e5cd4dc165587a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c013b182b64d41238e5cd4dc165587a0" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33521 } }
I20260927 10:57:46.096279 4356 raft_consensus.cc:388] T 00000000000000000000000000000000 P c013b182b64d41238e5cd4dc165587a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:46.096323 4356 raft_consensus.cc:482] T 00000000000000000000000000000000 P c013b182b64d41238e5cd4dc165587a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:46.096391 4356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c013b182b64d41238e5cd4dc165587a0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:46.097235 4356 raft_consensus.cc:504] T 00000000000000000000000000000000 P c013b182b64d41238e5cd4dc165587a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c013b182b64d41238e5cd4dc165587a0" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33521 } }
I20260927 10:57:46.097615 4356 leader_election.cc:302] T 00000000000000000000000000000000 P c013b182b64d41238e5cd4dc165587a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c013b182b64d41238e5cd4dc165587a0; no voters:
I20260927 10:57:46.097971 4356 leader_election.cc:288] T 00000000000000000000000000000000 P c013b182b64d41238e5cd4dc165587a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:46.098078 4366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c013b182b64d41238e5cd4dc165587a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:46.098313 4366 raft_consensus.cc:686] T 00000000000000000000000000000000 P c013b182b64d41238e5cd4dc165587a0 [term 1 LEADER]: Becoming Leader. State: Replica: c013b182b64d41238e5cd4dc165587a0, State: Running, Role: LEADER
I20260927 10:57:46.098680 4366 consensus_queue.cc:237] T 00000000000000000000000000000000 P c013b182b64d41238e5cd4dc165587a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c013b182b64d41238e5cd4dc165587a0" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33521 } }
I20260927 10:57:46.099265 4356 sys_catalog.cc:564] T 00000000000000000000000000000000 P c013b182b64d41238e5cd4dc165587a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:46.100373 4368 sys_catalog.cc:455] T 00000000000000000000000000000000 P c013b182b64d41238e5cd4dc165587a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c013b182b64d41238e5cd4dc165587a0. Latest consensus state: current_term: 1 leader_uuid: "c013b182b64d41238e5cd4dc165587a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c013b182b64d41238e5cd4dc165587a0" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33521 } } }
I20260927 10:57:46.100497 4368 sys_catalog.cc:458] T 00000000000000000000000000000000 P c013b182b64d41238e5cd4dc165587a0 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:46.100872 4375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:46.100850 4367 sys_catalog.cc:455] T 00000000000000000000000000000000 P c013b182b64d41238e5cd4dc165587a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c013b182b64d41238e5cd4dc165587a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c013b182b64d41238e5cd4dc165587a0" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 33521 } } }
I20260927 10:57:46.100982 4367 sys_catalog.cc:458] T 00000000000000000000000000000000 P c013b182b64d41238e5cd4dc165587a0 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:46.104120 4375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:46.109076 4375 catalog_manager.cc:1347] Generated new cluster ID: 012c79450a5f4069aab09c53480ce66a
I20260927 10:57:46.109135 4375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:46.115590 4375 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:46.115927 4375 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:46.127096 4375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c013b182b64d41238e5cd4dc165587a0: Generated new TSK 0
I20260927 10:57:46.127874 4375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:57:46.140002 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 4288
2026-09-27T10:57:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1383 ms)
[----------] 18 tests from AuthTokenIssuingTest (24992 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:46 dist-test-slave-tz51 krb5kdc[4387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:46 dist-test-slave-tz51 krb5kdc[4387](info): set up 2 sockets
Sep 27 10:57:46 dist-test-slave-tz51 krb5kdc[4387](info): commencing operation
krb5kdc: starting...
W20260927 10:57:47.208142 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:57:47 dist-test-slave-tz51 krb5kdc[4387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:57:47.330936 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:43679
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:37899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:43679,127.14.68.61:35095,127.14.68.60:41635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:47.441193 4400 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:57:47.441498 4400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:47.441555 4400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:47.441622 4400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:47.445487 4400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:47.445574 4400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:47.445612 4400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:47.445647 4400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:47.445678 4400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:47.450729 4400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:37899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:43679,127.14.68.61:35095,127.14.68.60:41635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:43679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:47.451783 4400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:47.452870 4400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:47.461505 4406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:47.461511 4405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:47.461525 4408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:47.461738 4400 server_base.cc:1034] running on GCE node
I20260927 10:57:47.462425 4400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:47.463557 4400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:47.464721 4400 hybrid_clock.cc:648] HybridClock initialized: now 1790506667464705 us; error 27 us; skew 500 ppm
Sep 27 10:57:47 dist-test-slave-tz51 krb5kdc[4387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506667, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:47.467512 4400 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:57:47.468376 4400 webserver.cc:460] Webserver started at http://127.14.68.62:37259/ using document root <none> and password file <none>
I20260927 10:57:47.468875 4400 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:47.468931 4400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:47.469089 4400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:47.471028 4400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "1ef11600541949a084a05116c0c23b02"
format_stamp: "Formatted at 2026-09-27 10:57:47 on dist-test-slave-tz51"
server_key: "96e8dbc6b720ea3df7ecf04d9fae97a5"
server_key_iv: "f4fb0c1b90021d28344b4e8199e3356f"
server_key_version: "encryptionkey@0"
I20260927 10:57:47.471529 4400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "1ef11600541949a084a05116c0c23b02"
format_stamp: "Formatted at 2026-09-27 10:57:47 on dist-test-slave-tz51"
server_key: "96e8dbc6b720ea3df7ecf04d9fae97a5"
server_key_iv: "f4fb0c1b90021d28344b4e8199e3356f"
server_key_version: "encryptionkey@0"
I20260927 10:57:47.475078 4400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260927 10:57:47.477421 4415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:47.478615 4400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:57:47.478744 4400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "1ef11600541949a084a05116c0c23b02"
format_stamp: "Formatted at 2026-09-27 10:57:47 on dist-test-slave-tz51"
server_key: "96e8dbc6b720ea3df7ecf04d9fae97a5"
server_key_iv: "f4fb0c1b90021d28344b4e8199e3356f"
server_key_version: "encryptionkey@0"
I20260927 10:57:47.478844 4400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:47.507462 4400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:47.510905 4400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:47.511114 4400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:47.518275 4400 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:43679
I20260927 10:57:47.518565 4467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:43679 every 8 connection(s)
I20260927 10:57:47.519248 4400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:47.522006 4468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:47.525676 4468 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43679 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:57:47.527143 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 4400
I20260927 10:57:47.527279 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:47.527612 14608 external_mini_cluster.cc:1428] Setting key bcc2f1ec9d0ac017ddc6da67b584bd8f
WARNING: no policy specified for kudu/127.14.68.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.61@KRBTEST.COM" created.
Sep 27 10:57:47 dist-test-slave-tz51 krb5kdc[4387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506667, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:57:47.545240 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.528014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:43679 (local address 127.0.0.1:59974)
0927 10:57:47.528556 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.528614 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:47.529672 (+ 1058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:47.530491 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:47.530498 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:47.530969 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:47.531358 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:47.531378 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.532642 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.532648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:47.534031 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:47.534047 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.534716 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.534723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:47.534866 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:47.535457 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:47.535482 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:47.539803 (+ 4321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:47.542247 (+ 2444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:47.542255 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:47.542264 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:47.542485 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:47.542855 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:47.542858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:47.542859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:47.542962 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:47.543820 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:47.543825 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:47.543948 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 10:57:47.544101 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 10:57:47.544364 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":138,"threads_started":1}
I20260927 10:57:47.545243 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.528014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59974 (local address 127.14.68.62:43679)
0927 10:57:47.528874 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:47.528882 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:47.528916 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:57:47.529987 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:47.529995 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:47.530262 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:47.530457 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:47.531518 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.532505 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.534168 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.534545 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.534664 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:47.539973 (+ 5309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:47.539979 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:47.539991 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:47.540014 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:47.542131 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.542596 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.542606 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.542610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.542763 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.543057 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.543064 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.543068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.543632 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:47.543843 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:47.544219 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 10:57:47.544486 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":765,"spinlock_wait_cycles":53504,"thread_start_us":138,"threads_started":1}
I20260927 10:57:47.547986 4468 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:57:47.548702 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.548470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:35095 (local address 127.0.0.1:44428)
0927 10:57:47.548582 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.548647 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.61:35095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 10:57:47.548897 4417 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.68.61:35095: connect: Connection refused (error 111)
I20260927 10:57:47.549674 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.549442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:35095 (local address 127.0.0.1:44444)
0927 10:57:47.549566 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.549617 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.61:35095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 10:57:47.549914 4468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.61:35095: Network error: Client connection negotiation failed: client connection to 127.14.68.61:35095: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.61.keytab.
Entry for principal kudu/127.14.68.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.61.keytab.
I20260927 10:57:47.555176 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.61:35095
--webserver_interface=127.14.68.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.61.keytab
--principal=kudu/127.14.68.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:37899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:43679,127.14.68.61:35095,127.14.68.60:41635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:57:47.599166 4468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } attempt: 1
I20260927 10:57:47.600134 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.599802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:35095 (local address 127.0.0.1:44452)
0927 10:57:47.599960 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.600023 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.61:35095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 10:57:47.600872 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.600653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:35095 (local address 127.0.0.1:44454)
0927 10:57:47.600768 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.600819 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.61:35095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 10:57:47.601027 4468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.61:35095: Network error: Client connection negotiation failed: client connection to 127.14.68.61:35095: connect: Connection refused (error 111)
W20260927 10:57:47.665764 4475 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:57:47.666142 4475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:47.666198 4475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:47.666272 4475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 10:57:47.669198 4468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } attempt: 2
I20260927 10:57:47.670178 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.669777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:35095 (local address 127.0.0.1:44458)
0927 10:57:47.669989 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.670095 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.61:35095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 10:57:47.670698 4475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:47.670768 4475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:47.670836 4475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:47.670892 4475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:47.670930 4475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:47.671025 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.670746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:35095 (local address 127.0.0.1:44474)
0927 10:57:47.670862 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.670944 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.61:35095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 10:57:47.671156 4468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.61:35095: Network error: Client connection negotiation failed: client connection to 127.14.68.61:35095: connect: Connection refused (error 111)
I20260927 10:57:47.676409 4475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:37899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:43679,127.14.68.61:35095,127.14.68.60:41635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.61.keytab
--principal=kudu/127.14.68.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.61:35095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.68.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:47.677536 4475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:47.678556 4475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:47.686988 4481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:47.687000 4483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:47.687009 4480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:47.687299 4475 server_base.cc:1034] running on GCE node
I20260927 10:57:47.687880 4475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:47.688923 4475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:47.690106 4475 hybrid_clock.cc:648] HybridClock initialized: now 1790506667690090 us; error 27 us; skew 500 ppm
Sep 27 10:57:47 dist-test-slave-tz51 krb5kdc[4387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506667, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:47.692756 4475 init.cc:377] Logged in from keytab as kudu/127.14.68.61@KRBTEST.COM (short username kudu)
I20260927 10:57:47.693606 4475 webserver.cc:460] Webserver started at http://127.14.68.61:36951/ using document root <none> and password file <none>
I20260927 10:57:47.694156 4475 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:47.694244 4475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:47.694475 4475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:47.696631 4475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-1/data/instance:
uuid: "1453604ca2804e98b0c092cca6d3666a"
format_stamp: "Formatted at 2026-09-27 10:57:47 on dist-test-slave-tz51"
server_key: "b7b8608b313e39e3878b588a9a74bf99"
server_key_iv: "3dad9aad634decb55c0cdbf972cecfa6"
server_key_version: "encryptionkey@0"
I20260927 10:57:47.697237 4475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-1/wal/instance:
uuid: "1453604ca2804e98b0c092cca6d3666a"
format_stamp: "Formatted at 2026-09-27 10:57:47 on dist-test-slave-tz51"
server_key: "b7b8608b313e39e3878b588a9a74bf99"
server_key_iv: "3dad9aad634decb55c0cdbf972cecfa6"
server_key_version: "encryptionkey@0"
I20260927 10:57:47.700884 4475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:57:47.703213 4490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:47.704638 4475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:47.704777 4475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-1/wal
uuid: "1453604ca2804e98b0c092cca6d3666a"
format_stamp: "Formatted at 2026-09-27 10:57:47 on dist-test-slave-tz51"
server_key: "b7b8608b313e39e3878b588a9a74bf99"
server_key_iv: "3dad9aad634decb55c0cdbf972cecfa6"
server_key_version: "encryptionkey@0"
I20260927 10:57:47.704883 4475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:47.723850 4475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:47.726979 4475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:47.727185 4475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:47.734464 4475 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.61:35095
I20260927 10:57:47.734784 4542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.61:35095 every 8 connection(s)
I20260927 10:57:47.735495 4475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-1/data/info.pb
I20260927 10:57:47.738418 4543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:47.741284 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 4475
I20260927 10:57:47.741436 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-1/wal/instance
I20260927 10:57:47.741735 14608 external_mini_cluster.cc:1428] Setting key 9d924aa11b1413c9ada172a0b05e95b3
I20260927 10:57:47.742533 4543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43679 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.68.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.60@KRBTEST.COM" created.
Sep 27 10:57:47 dist-test-slave-tz51 krb5kdc[4387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506667, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.61@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:57:47.756716 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.744788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59982 (local address 127.14.68.62:43679)
0927 10:57:47.744965 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:47.744972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:47.745757 (+ 785us) server_negotiation.cc:407] Connection header received
0927 10:57:47.746681 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:47.746686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:47.746759 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:47.746868 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:47.748087 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.748964 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.750161 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.750389 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.750471 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:47.753196 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:47.753200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:47.753203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:47.753235 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:47.754672 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.755175 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.755180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.755182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.755261 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.755598 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.755603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.755605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.755839 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:47.755943 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:47.756421 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 10:57:47.756540 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:57:47.757450 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.745131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:43679 (local address 127.0.0.1:59982)
0927 10:57:47.745595 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.745646 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:47.746477 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:47.747070 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:47.747077 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:47.747407 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:47.747933 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:47.747946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.749115 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.749119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:47.750046 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:47.750055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.750525 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.750528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:47.750718 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:47.751264 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:47.751279 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:47.753032 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:47.754800 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:47.754806 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:47.754819 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:47.755072 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:47.755353 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:47.755357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:47.755359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:47.755503 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:47.755941 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:47.755947 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:47.756187 (+ 240us) client_negotiation.cc:769] Sending connection context
0927 10:57:47.756367 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 10:57:47.756672 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":148,"threads_started":1}
I20260927 10:57:47.758771 4543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:57:47.762346 4468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } attempt: 3
Sep 27 10:57:47 dist-test-slave-tz51 krb5kdc[4387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506667, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.61@KRBTEST.COM for kudu/127.14.68.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.60.keytab.
Entry for principal kudu/127.14.68.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.60.keytab.
I20260927 10:57:47.767225 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.60:41635
--webserver_interface=127.14.68.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.60.keytab
--principal=kudu/127.14.68.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:37899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:43679,127.14.68.61:35095,127.14.68.60:41635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:57:47 dist-test-slave-tz51 krb5kdc[4387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506667, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for kudu/127.14.68.61@KRBTEST.COM
I20260927 10:57:47.769392 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.759302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:35095 (local address 127.0.0.1:44480)
0927 10:57:47.759411 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.759422 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:47.759479 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:47.760086 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:47.760088 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:47.760237 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:47.760338 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:47.760343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.761321 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.761327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:47.761950 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:47.761958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.762630 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.762634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:47.762708 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:47.763281 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:47.763293 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:47.764933 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:47.767494 (+ 2561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:47.767497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:47.767499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:47.767843 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:47.768190 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:47.768196 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:47.768201 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:47.768300 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:47.768844 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:47.768846 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:47.768929 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 10:57:47.769050 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 10:57:47.769195 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 10:57:47.770169 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.759406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.14.68.61:35095)
0927 10:57:47.759682 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:47.759691 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:47.759750 (+ 59us) server_negotiation.cc:407] Connection header received
0927 10:57:47.759792 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:47.759796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:47.759931 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:47.760072 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:47.760432 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.761205 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.762056 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.762419 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.762540 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:47.765066 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:47.765074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:47.765085 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:47.765104 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:47.767369 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.767963 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.767973 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.767978 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.768071 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.768405 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.768410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.768415 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.768731 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:47.768872 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:47.769800 (+ 928us) server_negotiation.cc:299] Negotiation successful
0927 10:57:47.769956 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":112,"threads_started":1}
I20260927 10:57:47.772531 4543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:57:47.773262 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.762772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:35095 (local address 127.0.0.1:44488)
0927 10:57:47.763258 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.763271 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:47.763376 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:47.763937 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:47.763940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:47.764066 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:47.764140 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:47.764145 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.765255 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.765260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:47.765872 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:47.765881 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.766572 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.766579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:47.766710 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:47.767180 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:47.767200 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:47.768891 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:47.771389 (+ 2498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:47.771392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:47.771395 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:47.771596 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:47.771999 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:47.772003 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:47.772005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:47.772091 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:47.772732 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:47.772735 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:47.772840 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 10:57:47.772944 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 10:57:47.773096 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 10:57:47.773272 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.762858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44488 (local address 127.14.68.61:35095)
0927 10:57:47.763148 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:47.763160 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:47.763349 (+ 189us) server_negotiation.cc:407] Connection header received
0927 10:57:47.763721 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:47.763724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:47.763797 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:47.763915 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:47.764277 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.765156 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.766032 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.766438 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.766551 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:47.769037 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:47.769043 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:47.769048 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:47.769067 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:47.771275 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.771769 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.771773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.771774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.771877 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.772462 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.772464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.772465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.772628 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:47.772741 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:47.773004 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 10:57:47.773107 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":84,"threads_started":1}
I20260927 10:57:47.773396 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.773076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:41635 (local address 127.0.0.1:36512)
0927 10:57:47.773208 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.773303 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 10:57:47.773589 4491 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.68.60:41635: connect: Connection refused (error 111)
I20260927 10:57:47.774533 4468 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:57:47.774775 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.774259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:41635 (local address 127.0.0.1:36516)
0927 10:57:47.774607 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.774677 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241}
W20260927 10:57:47.774955 4543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.60:41635: Network error: Client connection negotiation failed: client connection to 127.14.68.60:41635: connect: Connection refused (error 111)
I20260927 10:57:47.775336 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.774990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:41635 (local address 127.0.0.1:36520)
0927 10:57:47.775163 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.775233 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 10:57:47.776194 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.775855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:41635 (local address 127.0.0.1:36526)
0927 10:57:47.776039 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.776108 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260927 10:57:47.776366 4468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.60:41635: Network error: Client connection negotiation failed: client connection to 127.14.68.60:41635: connect: Connection refused (error 111)
I20260927 10:57:47.807540 4468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } attempt: 1
I20260927 10:57:47.808432 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.808116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:41635 (local address 127.0.0.1:36536)
0927 10:57:47.808258 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.808318 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 10:57:47.809262 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.809026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:41635 (local address 127.0.0.1:36542)
0927 10:57:47.809169 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.809213 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 10:57:47.809403 4468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.60:41635: Network error: Client connection negotiation failed: client connection to 127.14.68.60:41635: connect: Connection refused (error 111)
I20260927 10:57:47.824223 4543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } attempt: 1
I20260927 10:57:47.825134 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.824779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:41635 (local address 127.0.0.1:36554)
0927 10:57:47.824956 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.825033 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 10:57:47.825896 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.825667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:41635 (local address 127.0.0.1:36568)
0927 10:57:47.825779 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.825832 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 10:57:47.826063 4543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.60:41635: Network error: Client connection negotiation failed: client connection to 127.14.68.60:41635: connect: Connection refused (error 111)
I20260927 10:57:47.884590 4468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } attempt: 2
W20260927 10:57:47.884604 4550 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:57:47.884958 4550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:47.885016 4550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:47.885119 4550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 10:57:47.885512 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.885176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:41635 (local address 127.0.0.1:36580)
0927 10:57:47.885347 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.885419 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 10:57:47.886251 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.886027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:41635 (local address 127.0.0.1:36582)
0927 10:57:47.886137 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.886186 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 10:57:47.886394 4468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.60:41635: Network error: Client connection negotiation failed: client connection to 127.14.68.60:41635: connect: Connection refused (error 111)
W20260927 10:57:47.889667 4550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:47.889719 4550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:47.889758 4550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:47.889793 4550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:47.889828 4550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:47.894248 4543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } attempt: 2
I20260927 10:57:47.895052 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.894776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:41635 (local address 127.0.0.1:36590)
0927 10:57:47.894920 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.894981 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 10:57:47.895661 4550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:37899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:43679,127.14.68.61:35095,127.14.68.60:41635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.60.keytab
--principal=kudu/127.14.68.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.60:41635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.68.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:47.895815 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.895529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:41635 (local address 127.0.0.1:36600)
0927 10:57:47.895645 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.895751 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 10:57:47.895982 4543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.60:41635: Network error: Client connection negotiation failed: client connection to 127.14.68.60:41635: connect: Connection refused (error 111)
I20260927 10:57:47.896903 4550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:47.897934 4550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:47.906543 4559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:47.906534 4556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:47.906603 4550 server_base.cc:1034] running on GCE node
W20260927 10:57:47.906534 4557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:47.907298 4550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:47.908430 4550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:47.909610 4550 hybrid_clock.cc:648] HybridClock initialized: now 1790506667909577 us; error 44 us; skew 500 ppm
Sep 27 10:57:47 dist-test-slave-tz51 krb5kdc[4387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506667, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:47.912396 4550 init.cc:377] Logged in from keytab as kudu/127.14.68.60@KRBTEST.COM (short username kudu)
I20260927 10:57:47.913255 4550 webserver.cc:460] Webserver started at http://127.14.68.60:43185/ using document root <none> and password file <none>
I20260927 10:57:47.913815 4550 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:47.913905 4550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:47.914127 4550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:47.916169 4550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-2/data/instance:
uuid: "3367d9a1e2f64398b339dab03dea13c0"
format_stamp: "Formatted at 2026-09-27 10:57:47 on dist-test-slave-tz51"
server_key: "2a04037e8ca5b8fc53acccee830a70a8"
server_key_iv: "6b1369a6d9dffd0096a3643ae16455ce"
server_key_version: "encryptionkey@0"
I20260927 10:57:47.916733 4550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-2/wal/instance:
uuid: "3367d9a1e2f64398b339dab03dea13c0"
format_stamp: "Formatted at 2026-09-27 10:57:47 on dist-test-slave-tz51"
server_key: "2a04037e8ca5b8fc53acccee830a70a8"
server_key_iv: "6b1369a6d9dffd0096a3643ae16455ce"
server_key_version: "encryptionkey@0"
I20260927 10:57:47.920397 4550 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:57:47.922684 4566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:47.923905 4550 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:57:47.924029 4550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-2/wal
uuid: "3367d9a1e2f64398b339dab03dea13c0"
format_stamp: "Formatted at 2026-09-27 10:57:47 on dist-test-slave-tz51"
server_key: "2a04037e8ca5b8fc53acccee830a70a8"
server_key_iv: "6b1369a6d9dffd0096a3643ae16455ce"
server_key_version: "encryptionkey@0"
I20260927 10:57:47.924132 4550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:47.942765 4550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:47.945711 4550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:47.945922 4550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:47.953023 4550 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.60:41635
I20260927 10:57:47.953334 4618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.60:41635 every 8 connection(s)
I20260927 10:57:47.954074 4550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-2/data/info.pb
I20260927 10:57:47.954769 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 4550
I20260927 10:57:47.954891 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506389460539-14608-0/minicluster-data/master-2/wal/instance
I20260927 10:57:47.955170 14608 external_mini_cluster.cc:1428] Setting key 002e2954a68f92d67986e6c4a9205a82
I20260927 10:57:47.957227 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:47.961583 4619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43679 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:57:47 dist-test-slave-tz51 krb5kdc[4387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.60@KRBTEST.COM
Sep 27 10:57:47 dist-test-slave-tz51 krb5kdc[4387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506667, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.60@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:57:47.969796 4622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.956555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.14.68.60:41635)
0927 10:57:47.956968 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:47.956986 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:47.957028 (+ 42us) server_negotiation.cc:407] Connection header received
0927 10:57:47.957856 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:47.957878 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:47.958264 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:47.958614 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:47.959301 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.960835 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.961706 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.962351 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.962503 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:47.965114 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:47.965120 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:47.965131 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:47.965161 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:47.966890 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.967322 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.967327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.967330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.967437 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.967752 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.967761 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.967762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.968100 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:47.968301 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:47.968592 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 10:57:47.968890 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":125,"threads_started":1}
I20260927 10:57:47.973212 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.963495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59990 (local address 127.14.68.62:43679)
0927 10:57:47.963645 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:47.963650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:47.963965 (+ 315us) server_negotiation.cc:407] Connection header received
0927 10:57:47.964113 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:47.964116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:47.964174 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:47.964256 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:47.964941 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.965668 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.966654 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.967014 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.967127 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:47.969236 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:47.969240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:47.969243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:47.969268 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:47.971322 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.971909 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.971912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.971913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.971985 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.972281 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.972285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.972288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.972486 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:47.972599 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:47.972940 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 10:57:47.973050 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:57:47.973237 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.963464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:43679 (local address 127.0.0.1:59990)
0927 10:57:47.963842 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.963879 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:47.964007 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:47.964302 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:47.964307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:47.964674 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:47.964822 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:47.964828 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.965795 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.965797 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:47.966525 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:47.966535 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.967127 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.967130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:47.967245 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:47.967809 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:47.967820 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:47.969079 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:47.971428 (+ 2349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:47.971435 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:47.971441 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:47.971811 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:47.972083 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:47.972087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:47.972089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:47.972180 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:47.972618 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:47.972624 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:47.972729 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 10:57:47.972858 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 10:57:47.973036 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"spinlock_wait_cycles":2944,"thread_start_us":118,"threads_started":1}
I20260927 10:57:47.974184 4619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:57:47 dist-test-slave-tz51 krb5kdc[4387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
Sep 27 10:57:47 dist-test-slave-tz51 krb5kdc[4387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506667, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.60@KRBTEST.COM for kudu/127.14.68.61@KRBTEST.COM
I20260927 10:57:47.981208 4624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.972129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59994 (local address 127.14.68.62:43679)
0927 10:57:47.972492 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:47.972499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:47.972509 (+ 10us) server_negotiation.cc:407] Connection header received
0927 10:57:47.972538 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:47.972539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:47.972572 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:47.972674 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:47.973244 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.974050 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.974941 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.975179 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.975288 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:47.977844 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:47.977846 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:47.977848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:47.977867 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:47.979389 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.979846 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.979853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.979857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.979944 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.980248 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.980254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.980259 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.980497 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:47.980728 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:47.980842 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 10:57:47.980960 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":119,"threads_started":1}
I20260927 10:57:47.982903 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.974666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:35095 (local address 127.0.0.1:44498)
0927 10:57:47.974800 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.974811 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:47.974893 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:47.975172 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:47.975175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:47.975338 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:47.975421 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:47.975426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.976311 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.976317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:47.977022 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:47.977037 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.977584 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.977588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:47.977699 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:47.978198 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:47.978215 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:47.979431 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:47.981347 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:47.981350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:47.981352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:47.981541 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:47.981834 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:47.981837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:47.981839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:47.981926 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:47.982362 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:47.982366 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:47.982461 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 10:57:47.982595 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 10:57:47.982730 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 10:57:47.982934 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.974802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44498 (local address 127.14.68.61:35095)
0927 10:57:47.974954 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:47.974961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:47.974981 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:57:47.975027 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:47.975029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:47.975075 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:47.975160 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:47.975538 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.976205 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.977177 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.977470 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.977554 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:47.979588 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:47.979590 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:47.979592 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:47.979616 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:47.981219 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.981636 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.981639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.981641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.981729 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.982003 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.982005 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.982006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.982263 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:47.982360 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:47.982651 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 10:57:47.982769 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:57:47.984040 4619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:57:47.985585 4468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } attempt: 3
I20260927 10:57:47.987114 4543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } attempt: 3
Sep 27 10:57:47 dist-test-slave-tz51 krb5kdc[4387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.61@KRBTEST.COM
Sep 27 10:57:47 dist-test-slave-tz51 krb5kdc[4387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506667, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.60@KRBTEST.COM for kudu/127.14.68.60@KRBTEST.COM
I20260927 10:57:47.992327 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.982419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44510 (local address 127.14.68.61:35095)
0927 10:57:47.982540 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:47.982547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:47.982561 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:57:47.982690 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:47.982693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:47.982741 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:47.982822 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:47.983476 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.984206 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.985133 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.985372 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.985479 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:47.988176 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:47.988182 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:47.988188 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:47.988212 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:47.990103 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.990606 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.990610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.990612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.990693 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.990961 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.990964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.990966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.991158 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:47.991347 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:47.992048 (+ 701us) server_negotiation.cc:299] Negotiation successful
0927 10:57:47.992155 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 10:57:47 dist-test-slave-tz51 krb5kdc[4387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506667, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for kudu/127.14.68.60@KRBTEST.COM
Sep 27 10:57:47 dist-test-slave-tz51 krb5kdc[4387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506667, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.61@KRBTEST.COM for kudu/127.14.68.60@KRBTEST.COM
I20260927 10:57:47.993777 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.984739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:41635 (local address 127.0.0.1:36618)
0927 10:57:47.984845 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.984859 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:47.985019 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:47.985293 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:47.985296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:47.985451 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:47.985507 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:47.985512 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.986467 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.986468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:47.986935 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:47.986940 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.987552 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.987555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:47.987629 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:47.988105 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:47.988121 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:47.989747 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:47.992276 (+ 2529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:47.992279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:47.992281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:47.992502 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:47.992765 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:47.992769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:47.992771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:47.992866 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:47.993298 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:47.993300 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:47.993378 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 10:57:47.993507 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 10:57:47.993635 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 10:57:47.994426 4622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.984887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.14.68.60:41635)
0927 10:57:47.984964 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:47.984968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:47.984979 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:57:47.985158 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:47.985161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:47.985197 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:47.985282 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:47.985663 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.986381 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.987042 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.987352 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.987443 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:47.989919 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:47.989922 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:47.989924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:47.989944 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:47.992178 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.992596 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.992599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.992600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.992668 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.992957 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.992962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.992963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.993168 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:47.993259 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:47.994146 (+ 887us) server_negotiation.cc:299] Negotiation successful
0927 10:57:47.994263 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25,"spinlock_wait_cycles":7424}
I20260927 10:57:47.996528 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:57:47.997056 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.985929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:41635 (local address 127.0.0.1:36632)
0927 10:57:47.986102 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.986114 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:47.986179 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:47.987746 (+ 1567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:47.987750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:47.988197 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:47.988286 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:47.988291 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.989485 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.989491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:47.990312 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:47.990325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.991037 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.991041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:47.991144 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:47.991552 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:47.991569 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:47.992965 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:47.995489 (+ 2524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:47.995492 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:47.995493 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:47.995791 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:47.996104 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:47.996107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:47.996108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:47.996186 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:47.996572 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:47.996575 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:47.996690 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 10:57:47.996772 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 10:57:47.996892 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260927 10:57:47.997615 4625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.986063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36632 (local address 127.14.68.60:41635)
0927 10:57:47.987461 (+ 1398us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:47.987465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:47.987478 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:57:47.987525 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:47.987528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:47.987574 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:47.988372 (+ 798us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:47.988510 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.989357 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.990463 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.990895 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.991366 (+ 471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:47.993119 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:47.993125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:47.993131 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:47.993166 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:47.995375 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.995914 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.995919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.995921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.995999 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.996281 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.996284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.996286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.996452 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:47.997271 (+ 819us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:47.997343 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 10:57:47.997440 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":989,"spinlock_wait_cycles":896,"thread_start_us":100,"threads_started":1}
I20260927 10:57:47.998070 4619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3367d9a1e2f64398b339dab03dea13c0: Bootstrap starting.
I20260927 10:57:47.998509 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.987476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:41635 (local address 127.0.0.1:36642)
0927 10:57:47.987630 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:47.987643 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:47.987741 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:47.989011 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:47.989014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:47.989179 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:47.989247 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:47.989252 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.990267 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.990271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:47.991065 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:47.991073 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.991631 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.991635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:47.991754 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:47.992169 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:47.992181 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:47.993814 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:47.996768 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:47.996771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:47.996773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:47.997013 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:47.997330 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:47.997332 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:47.997334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:47.997418 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:47.998053 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:47.998056 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:47.998154 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 10:57:47.998239 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 10:57:47.998362 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 10:57:47.998973 4626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:47.987923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36642 (local address 127.14.68.60:41635)
0927 10:57:47.988758 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:47.988769 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:47.988784 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:57:47.988819 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:47.988821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:47.988861 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:47.988997 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:47.989366 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.990152 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.991200 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:47.991506 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:47.991939 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:47.994695 (+ 2756us) spinlock_profiling.cc:243] Waited 1.93 ms on lock 0x129b8cb8. stack: 00007f8c2d21cc78 00007f8c2d21d1a9 00007f8c2cddf1d7 00000000005984a6 00000000005984f9 000000000059c488 00007f8c2d2192ab 00007f8c2d2189da 00007f8c318f10c4 00007f8c318f9fc5 00007f8c318fc29d 00007f8c318f8422 00007f8c318b1219 00007f8c318b1615 00007f8c318c70d3 00007f8c318ca54b
0927 10:57:47.994784 (+ 89us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:47.994788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:47.994791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:47.994812 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:47.996627 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.997140 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.997144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.997146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.997224 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:47.997534 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:47.997539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:47.997540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:47.997746 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:47.998648 (+ 902us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:47.998719 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 10:57:47.998817 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"spinlock_wait_cycles":4436736,"thread_start_us":122,"threads_started":1}
I20260927 10:57:47.998214 4442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1ef11600541949a084a05116c0c23b02"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59994
I20260927 10:57:48.000499 4468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02: Bootstrap starting.
I20260927 10:57:48.001121 4619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3367d9a1e2f64398b339dab03dea13c0: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:48.001945 4619 log.cc:824] T 00000000000000000000000000000000 P 3367d9a1e2f64398b339dab03dea13c0: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:48.002211 4543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1453604ca2804e98b0c092cca6d3666a: Bootstrap starting.
I20260927 10:57:48.002754 4468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:48.003513 4468 log.cc:824] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:48.004309 4619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3367d9a1e2f64398b339dab03dea13c0: No bootstrap required, opened a new log
I20260927 10:57:48.004534 4543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1453604ca2804e98b0c092cca6d3666a: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:48.005165 4543 log.cc:824] T 00000000000000000000000000000000 P 1453604ca2804e98b0c092cca6d3666a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:48.006076 4468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02: No bootstrap required, opened a new log
I20260927 10:57:48.007162 4543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1453604ca2804e98b0c092cca6d3666a: No bootstrap required, opened a new log
I20260927 10:57:48.009068 4468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef11600541949a084a05116c0c23b02" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43679 } } peers { permanent_uuid: "1453604ca2804e98b0c092cca6d3666a" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } } peers { permanent_uuid: "3367d9a1e2f64398b339dab03dea13c0" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } }
I20260927 10:57:48.009132 4619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3367d9a1e2f64398b339dab03dea13c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef11600541949a084a05116c0c23b02" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43679 } } peers { permanent_uuid: "1453604ca2804e98b0c092cca6d3666a" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } } peers { permanent_uuid: "3367d9a1e2f64398b339dab03dea13c0" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } }
I20260927 10:57:48.009306 4468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ef11600541949a084a05116c0c23b02, State: Initialized, Role: FOLLOWER
I20260927 10:57:48.009320 4619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3367d9a1e2f64398b339dab03dea13c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3367d9a1e2f64398b339dab03dea13c0, State: Initialized, Role: FOLLOWER
I20260927 10:57:48.009645 4619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3367d9a1e2f64398b339dab03dea13c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef11600541949a084a05116c0c23b02" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43679 } } peers { permanent_uuid: "1453604ca2804e98b0c092cca6d3666a" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } } peers { permanent_uuid: "3367d9a1e2f64398b339dab03dea13c0" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } }
I20260927 10:57:48.009645 4468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef11600541949a084a05116c0c23b02" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43679 } } peers { permanent_uuid: "1453604ca2804e98b0c092cca6d3666a" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } } peers { permanent_uuid: "3367d9a1e2f64398b339dab03dea13c0" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } }
I20260927 10:57:48.010354 4630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef11600541949a084a05116c0c23b02" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43679 } } peers { permanent_uuid: "1453604ca2804e98b0c092cca6d3666a" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } } peers { permanent_uuid: "3367d9a1e2f64398b339dab03dea13c0" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } } }
I20260927 10:57:48.010406 4631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3367d9a1e2f64398b339dab03dea13c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef11600541949a084a05116c0c23b02" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43679 } } peers { permanent_uuid: "1453604ca2804e98b0c092cca6d3666a" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } } peers { permanent_uuid: "3367d9a1e2f64398b339dab03dea13c0" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } } }
I20260927 10:57:48.010926 4619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3367d9a1e2f64398b339dab03dea13c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:48.011144 4630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:57:48.010926 4468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:48.011025 4442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 10:57:48.011294 4442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:48.011322 4543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1453604ca2804e98b0c092cca6d3666a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef11600541949a084a05116c0c23b02" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43679 } } peers { permanent_uuid: "1453604ca2804e98b0c092cca6d3666a" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } } peers { permanent_uuid: "3367d9a1e2f64398b339dab03dea13c0" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } }
I20260927 10:57:48.011543 4543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1453604ca2804e98b0c092cca6d3666a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1453604ca2804e98b0c092cca6d3666a, State: Initialized, Role: FOLLOWER
I20260927 10:57:48.011751 4631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3367d9a1e2f64398b339dab03dea13c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:57:48.011871 4543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1453604ca2804e98b0c092cca6d3666a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef11600541949a084a05116c0c23b02" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43679 } } peers { permanent_uuid: "1453604ca2804e98b0c092cca6d3666a" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } } peers { permanent_uuid: "3367d9a1e2f64398b339dab03dea13c0" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } }
I20260927 10:57:48.012578 4637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1453604ca2804e98b0c092cca6d3666a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef11600541949a084a05116c0c23b02" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43679 } } peers { permanent_uuid: "1453604ca2804e98b0c092cca6d3666a" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } } peers { permanent_uuid: "3367d9a1e2f64398b339dab03dea13c0" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } } }
I20260927 10:57:48.012643 4442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef11600541949a084a05116c0c23b02" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43679 } } peers { permanent_uuid: "1453604ca2804e98b0c092cca6d3666a" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } } peers { permanent_uuid: "3367d9a1e2f64398b339dab03dea13c0" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } }
I20260927 10:57:48.012825 4637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1453604ca2804e98b0c092cca6d3666a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:57:48.013191 4543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1453604ca2804e98b0c092cca6d3666a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:48.013679 4442 leader_election.cc:288] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02 [CANDIDATE]: Term 1 election: Requested vote from peers 1453604ca2804e98b0c092cca6d3666a (127.14.68.61:35095), 3367d9a1e2f64398b339dab03dea13c0 (127.14.68.60:41635)
W20260927 10:57:48.014312 4421 tablet.cc:2369] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:57:48.016559 4517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ef11600541949a084a05116c0c23b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1453604ca2804e98b0c092cca6d3666a"
W20260927 10:57:48.016649 4643 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3367d9a1e2f64398b339dab03dea13c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:57:48.016785 4643 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:57:48.016812 4517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1453604ca2804e98b0c092cca6d3666a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:48.018196 4517 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1453604ca2804e98b0c092cca6d3666a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ef11600541949a084a05116c0c23b02 in term 1.
I20260927 10:57:48.018652 4417 leader_election.cc:302] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1453604ca2804e98b0c092cca6d3666a, 1ef11600541949a084a05116c0c23b02; no voters:
I20260927 10:57:48.018656 4593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ef11600541949a084a05116c0c23b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3367d9a1e2f64398b339dab03dea13c0"
I20260927 10:57:48.018905 4593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3367d9a1e2f64398b339dab03dea13c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:48.019071 4630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:48.019359 4630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02 [term 1 LEADER]: Becoming Leader. State: Replica: 1ef11600541949a084a05116c0c23b02, State: Running, Role: LEADER
I20260927 10:57:48.019840 4630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef11600541949a084a05116c0c23b02" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43679 } } peers { permanent_uuid: "1453604ca2804e98b0c092cca6d3666a" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } } peers { permanent_uuid: "3367d9a1e2f64398b339dab03dea13c0" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } }
I20260927 10:57:48.016654 4475 master_runner.cc:186] Error getting master registration for 127.14.68.62:43679: OK, instance_id { permanent_uuid: "1ef11600541949a084a05116c0c23b02" instance_seqno: 1790506667516229 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:57:48.020195 4475 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:57:48.020221 4593 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3367d9a1e2f64398b339dab03dea13c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ef11600541949a084a05116c0c23b02 in term 1.
I20260927 10:57:48.022013 4630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ef11600541949a084a05116c0c23b02. Latest consensus state: current_term: 1 leader_uuid: "1ef11600541949a084a05116c0c23b02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef11600541949a084a05116c0c23b02" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43679 } } peers { permanent_uuid: "1453604ca2804e98b0c092cca6d3666a" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } } peers { permanent_uuid: "3367d9a1e2f64398b339dab03dea13c0" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } } }
I20260927 10:57:48.022189 4630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:48.022601 4663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260927 10:57:48.023211 4655 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1453604ca2804e98b0c092cca6d3666a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:57:48.023291 4655 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:57:48.026052 4663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:48.022495 4550 master_runner.cc:186] Error getting master registration for 127.14.68.62:43679: OK, instance_id { permanent_uuid: "1ef11600541949a084a05116c0c23b02" instance_seqno: 1790506667516229 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:57:48.026420 4550 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:57:48.041533 4593 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3367d9a1e2f64398b339dab03dea13c0 [term 1 FOLLOWER]: Refusing update from remote peer 1ef11600541949a084a05116c0c23b02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:48.042160 4630 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3367d9a1e2f64398b339dab03dea13c0" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:48.045971 4631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3367d9a1e2f64398b339dab03dea13c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ef11600541949a084a05116c0c23b02. Latest consensus state: current_term: 1 leader_uuid: "1ef11600541949a084a05116c0c23b02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef11600541949a084a05116c0c23b02" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43679 } } peers { permanent_uuid: "1453604ca2804e98b0c092cca6d3666a" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } } peers { permanent_uuid: "3367d9a1e2f64398b339dab03dea13c0" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } } }
I20260927 10:57:48.046110 4631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3367d9a1e2f64398b339dab03dea13c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:57:48.046552 4517 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1453604ca2804e98b0c092cca6d3666a [term 1 FOLLOWER]: Refusing update from remote peer 1ef11600541949a084a05116c0c23b02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:48.047065 4630 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1453604ca2804e98b0c092cca6d3666a" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:48.050752 4637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1453604ca2804e98b0c092cca6d3666a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ef11600541949a084a05116c0c23b02. Latest consensus state: current_term: 1 leader_uuid: "1ef11600541949a084a05116c0c23b02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef11600541949a084a05116c0c23b02" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43679 } } peers { permanent_uuid: "1453604ca2804e98b0c092cca6d3666a" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } } peers { permanent_uuid: "3367d9a1e2f64398b339dab03dea13c0" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } } }
I20260927 10:57:48.050874 4637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1453604ca2804e98b0c092cca6d3666a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:57:48.056448 4624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:48.023781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60008 (local address 127.14.68.62:43679)
0927 10:57:48.023892 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:48.023897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:48.036146 (+ 12249us) server_negotiation.cc:407] Connection header received
0927 10:57:48.036342 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:48.036345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:48.036390 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:48.036453 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:48.038335 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:48.039168 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:48.040814 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:48.041185 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:48.041274 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:48.051214 (+ 9940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:48.051219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:48.051223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:48.051258 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:48.053151 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:48.054873 (+ 1722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:48.054878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:48.054880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:48.054991 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:48.055319 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:48.055323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:48.055325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:48.055574 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:48.055744 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:48.056126 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 10:57:48.056275 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 10:57:48.060985 4656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ef11600541949a084a05116c0c23b02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef11600541949a084a05116c0c23b02" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43679 } } peers { permanent_uuid: "1453604ca2804e98b0c092cca6d3666a" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } } peers { permanent_uuid: "3367d9a1e2f64398b339dab03dea13c0" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } } }
I20260927 10:57:48.061164 4656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:48.061904 4630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ef11600541949a084a05116c0c23b02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef11600541949a084a05116c0c23b02" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43679 } } peers { permanent_uuid: "1453604ca2804e98b0c092cca6d3666a" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } } peers { permanent_uuid: "3367d9a1e2f64398b339dab03dea13c0" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } } }
I20260927 10:57:48.062001 4630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:48.062229 4631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3367d9a1e2f64398b339dab03dea13c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ef11600541949a084a05116c0c23b02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef11600541949a084a05116c0c23b02" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43679 } } peers { permanent_uuid: "1453604ca2804e98b0c092cca6d3666a" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } } peers { permanent_uuid: "3367d9a1e2f64398b339dab03dea13c0" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } } }
I20260927 10:57:48.062352 4631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3367d9a1e2f64398b339dab03dea13c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:57:48.062940 4637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1453604ca2804e98b0c092cca6d3666a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ef11600541949a084a05116c0c23b02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef11600541949a084a05116c0c23b02" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 43679 } } peers { permanent_uuid: "1453604ca2804e98b0c092cca6d3666a" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 35095 } } peers { permanent_uuid: "3367d9a1e2f64398b339dab03dea13c0" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 41635 } } }
I20260927 10:57:48.063059 4637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1453604ca2804e98b0c092cca6d3666a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:57:48.063140 4679 mvcc.cc:205] Tried to move back new op lower bound from 7333915312291016704 to 7333915312214310912. Current Snapshot: MvccSnapshot[applied={T|T < 7333915312291016704}]
I20260927 10:57:48.065016 4680 mvcc.cc:205] Tried to move back new op lower bound from 7333915312291016704 to 7333915312214310912. Current Snapshot: MvccSnapshot[applied={T|T < 7333915312291016704}]
I20260927 10:57:48.065253 4663 catalog_manager.cc:1347] Generated new cluster ID: d8c414b0a1c44507bf1cb9400ea4ae13
I20260927 10:57:48.065366 4663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:48.065675 4682 mvcc.cc:205] Tried to move back new op lower bound from 7333915312291016704 to 7333915312214310912. Current Snapshot: MvccSnapshot[applied={T|T < 7333915312291016704}]
I20260927 10:57:48.076571 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:48.024716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44522 (local address 127.14.68.61:35095)
0927 10:57:48.024825 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:48.024830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:48.040555 (+ 15725us) server_negotiation.cc:407] Connection header received
0927 10:57:48.041914 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:48.041917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:48.041961 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:48.042007 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:48.050250 (+ 8243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:48.051028 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:48.058392 (+ 7364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:48.058804 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:48.058899 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:48.071839 (+ 12940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:48.071844 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:48.071847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:48.071880 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:48.073614 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:48.074510 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:48.074513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:48.074515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:48.074615 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:48.075334 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:48.075337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:48.075339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:48.075580 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:48.075686 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:48.076272 (+ 586us) server_negotiation.cc:299] Negotiation successful
0927 10:57:48.076402 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:57:48.078565 4626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:48.024564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36648 (local address 127.14.68.60:41635)
0927 10:57:48.024676 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:48.024680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:48.040746 (+ 16066us) server_negotiation.cc:407] Connection header received
0927 10:57:48.042144 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:48.042147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:48.042192 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:48.042237 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:48.046350 (+ 4113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:48.047817 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:48.049161 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:48.049526 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:48.049609 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:48.062689 (+ 13080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:48.062693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:48.062696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:48.062732 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:48.064640 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:48.074111 (+ 9471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:48.074115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:48.074117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:48.074208 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:48.074910 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:48.074914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:48.074916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:48.075139 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:48.075257 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:48.078279 (+ 3022us) server_negotiation.cc:299] Negotiation successful
0927 10:57:48.078407 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:57:48.099843 4663 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:48.104445 4663 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:48.123293 4663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ef11600541949a084a05116c0c23b02: Generated new TSK 0
I20260927 10:57:48.124250 4663 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:57:48.156221 4548 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:48.153165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44526 (local address 127.14.68.61:35095)
0927 10:57:48.153292 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:48.153297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:48.155117 (+ 1820us) server_negotiation.cc:407] Connection header received
0927 10:57:48.155318 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:48.155321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:48.155374 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:48.155427 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:48.156098 (+ 671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44526: BlockingRecv error: recv got EOF from 127.0.0.1:44526 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 10:57:48.157593 4624 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:48.153944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60024 (local address 127.14.68.62:43679)
0927 10:57:48.154051 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:48.154056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:48.156696 (+ 2640us) server_negotiation.cc:407] Connection header received
0927 10:57:48.156869 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:48.156873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:48.156921 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:48.156974 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:48.157464 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60024: BlockingRecv error: recv got EOF from 127.0.0.1:60024 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 10:57:48.159003 4626 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:48.154540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36664 (local address 127.14.68.60:41635)
0927 10:57:48.154652 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:48.154657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:48.158048 (+ 3391us) server_negotiation.cc:407] Connection header received
0927 10:57:48.158216 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:48.158220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:48.158266 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:48.158316 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:48.158897 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36664: BlockingRecv error: recv got EOF from 127.0.0.1:36664 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 10:57:48.163352 4548 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:48.161842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44542 (local address 127.14.68.61:35095)
0927 10:57:48.161935 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:48.161938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:48.162370 (+ 432us) server_negotiation.cc:407] Connection header received
0927 10:57:48.162586 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:48.162589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:48.162658 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:48.162709 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:48.163257 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44542: BlockingRecv error: recv got EOF from 127.0.0.1:44542 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 10:57:48.168305 4548 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:48.166860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44556 (local address 127.14.68.61:35095)
0927 10:57:48.166953 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:48.166957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:48.167415 (+ 458us) server_negotiation.cc:407] Connection header received
0927 10:57:48.167612 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:48.167616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:48.167664 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:48.167735 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:48.168204 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44556: BlockingRecv error: recv got EOF from 127.0.0.1:44556 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 10:57:48.174050 4548 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:48.172449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.14.68.61:35095)
0927 10:57:48.172582 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:48.172586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:48.173142 (+ 556us) server_negotiation.cc:407] Connection header received
0927 10:57:48.173327 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:48.173331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:48.173378 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:48.173429 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:48.173946 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44566: BlockingRecv error: recv got EOF from 127.0.0.1:44566 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 10:57:48.182346 4548 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:48.180460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44574 (local address 127.14.68.61:35095)
0927 10:57:48.180623 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:48.180627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:48.181186 (+ 559us) server_negotiation.cc:407] Connection header received
0927 10:57:48.181461 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:48.181466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:48.181531 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:48.181590 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:48.182221 (+ 631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44574: BlockingRecv error: recv got EOF from 127.0.0.1:44574 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260927 10:57:48.194448 4548 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:48.192884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44580 (local address 127.14.68.61:35095)
0927 10:57:48.192996 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:48.193000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:48.193531 (+ 531us) server_negotiation.cc:407] Connection header received
0927 10:57:48.193727 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:48.193731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:48.193783 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:48.193833 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:48.194338 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44580: BlockingRecv error: recv got EOF from 127.0.0.1:44580 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 10:57:48.214572 4548 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:48.212860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44586 (local address 127.14.68.61:35095)
0927 10:57:48.212968 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:48.212973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:48.213528 (+ 555us) server_negotiation.cc:407] Connection header received
0927 10:57:48.213808 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:48.213812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:48.213863 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:48.213914 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:48.214442 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44586: BlockingRecv error: recv got EOF from 127.0.0.1:44586 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 10:57:48.250476 4548 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:48.249194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44590 (local address 127.14.68.61:35095)
0927 10:57:48.249329 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:48.249335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:48.249530 (+ 195us) server_negotiation.cc:407] Connection header received
0927 10:57:48.249711 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:48.249719 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:48.249768 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:48.249860 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:48.250347 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44590: BlockingRecv error: recv got EOF from 127.0.0.1:44590 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 10:57:48.317796 4548 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:48.316712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44596 (local address 127.14.68.61:35095)
0927 10:57:48.316851 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:48.316854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:48.316986 (+ 132us) server_negotiation.cc:407] Connection header received
0927 10:57:48.317115 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:48.317117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:48.317167 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:48.317252 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:48.317655 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44596: BlockingRecv error: recv got EOF from 127.0.0.1:44596 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 10:57:48.449221 4548 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:48.448015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44600 (local address 127.14.68.61:35095)
0927 10:57:48.448272 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:48.448277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:48.448300 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:57:48.448471 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:48.448475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:48.448526 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:48.448610 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:48.449122 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44600: BlockingRecv error: recv got EOF from 127.0.0.1:44600 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
W20260927 10:57:48.708560 4548 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:48.707472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44602 (local address 127.14.68.61:35095)
0927 10:57:48.707624 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:48.707627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:48.707641 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:57:48.707774 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:48.707778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:48.707836 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:48.707945 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:48.708441 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44602: BlockingRecv error: recv got EOF from 127.0.0.1:44602 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 10:57:49.018198 4643 catalog_manager.cc:1259] Loaded cluster ID: d8c414b0a1c44507bf1cb9400ea4ae13
I20260927 10:57:49.018292 4643 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3367d9a1e2f64398b339dab03dea13c0: loading cluster ID for follower catalog manager: success
I20260927 10:57:49.019977 4643 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3367d9a1e2f64398b339dab03dea13c0: acquiring CA information for follower catalog manager: success
I20260927 10:57:49.021180 4643 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3367d9a1e2f64398b339dab03dea13c0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:57:49.024564 4655 catalog_manager.cc:1259] Loaded cluster ID: d8c414b0a1c44507bf1cb9400ea4ae13
I20260927 10:57:49.024647 4655 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1453604ca2804e98b0c092cca6d3666a: loading cluster ID for follower catalog manager: success
I20260927 10:57:49.026243 4655 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1453604ca2804e98b0c092cca6d3666a: acquiring CA information for follower catalog manager: success
I20260927 10:57:49.027398 4655 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1453604ca2804e98b0c092cca6d3666a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:57:49.226969 4751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:49.222855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44618 (local address 127.14.68.61:35095)
0927 10:57:49.223079 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:49.223083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:49.223098 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:57:49.223304 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:49.223307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:49.223437 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:49.223552 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:57:49.223862 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:49.224393 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:49.225394 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:49.225629 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:49.225738 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:49.226071 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:57:49.226516 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:57:49.226657 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:49.226694 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 10:57:49.226749 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":82,"threads_started":1}
I20260927 10:57:49.233623 4757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:49.230069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36674 (local address 127.14.68.60:41635)
0927 10:57:49.230315 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:49.230318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:49.230331 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:57:49.230446 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:49.230448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:49.230565 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:49.230697 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:57:49.230870 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:49.231368 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:49.232247 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:49.232462 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:49.232560 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:49.232810 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:57:49.233220 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:57:49.233316 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:49.233366 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 10:57:49.233412 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":76,"threads_started":1}
I20260927 10:57:49.240285 4751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:49.236435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44622 (local address 127.14.68.61:35095)
0927 10:57:49.236568 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:49.236572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:49.236805 (+ 233us) server_negotiation.cc:407] Connection header received
0927 10:57:49.236996 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:49.236999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:49.237062 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:49.237149 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:57:49.237413 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:49.238048 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:49.239134 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:49.239474 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:49.239567 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:49.239840 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:57:49.239923 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:57:49.239995 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:49.240099 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 10:57:49.240164 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:57:49.240813 4757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:49.236825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36680 (local address 127.14.68.60:41635)
0927 10:57:49.236942 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:49.236948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:49.237182 (+ 234us) server_negotiation.cc:407] Connection header received
0927 10:57:49.237350 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:49.237356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:49.237411 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:49.237476 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:57:49.237801 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:49.238517 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:49.239491 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:49.239833 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:49.239918 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:49.240307 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:57:49.240414 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:57:49.240471 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:49.240619 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 10:57:49.240675 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:57:49.241457 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:49.236443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60026 (local address 127.14.68.62:43679)
0927 10:57:49.236690 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:49.236694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:49.236976 (+ 282us) server_negotiation.cc:407] Connection header received
0927 10:57:49.237075 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:49.237078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:49.237233 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:49.237343 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:57:49.237556 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:49.238316 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:49.239400 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:49.239788 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:49.239905 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:49.240250 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:57:49.240587 (+ 337us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:57:49.240674 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:49.240972 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 10:57:49.241038 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
I20260927 10:57:49.248126 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 4400
I20260927 10:57:49.254168 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 4475
I20260927 10:57:49.260228 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 4550
2026-09-27T10:57:49Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3122 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:49 dist-test-slave-tz51 krb5kdc[4768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:49 dist-test-slave-tz51 krb5kdc[4768](info): set up 2 sockets
Sep 27 10:57:49 dist-test-slave-tz51 krb5kdc[4768](info): commencing operation
krb5kdc: starting...
W20260927 10:57:50.304078 14608 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/test-user.keytab.
Sep 27 10:57:50 dist-test-slave-tz51 krb5kdc[4768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.68.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
Entry for principal kudu/127.14.68.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab.
I20260927 10:57:50.424006 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:41243
--webserver_interface=127.14.68.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:43721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:41243,127.14.68.61:42369,127.14.68.60:33573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:50.532028 4781 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:57:50.532312 4781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:50.532335 4781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:50.532415 4781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:50.536415 4781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:50.536484 4781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:50.536502 4781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:50.536520 4781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:50.536535 4781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:50.541839 4781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:43721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:41243,127.14.68.61:42369,127.14.68.60:33573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.62.keytab
--principal=kudu/127.14.68.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.62:41243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.68.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:50.542929 4781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:50.543982 4781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:50.552335 4786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:50.552353 4787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:50.552335 4789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:50.552590 4781 server_base.cc:1034] running on GCE node
I20260927 10:57:50.553179 4781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:50.554272 4781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:50.555429 4781 hybrid_clock.cc:648] HybridClock initialized: now 1790506670555402 us; error 44 us; skew 500 ppm
Sep 27 10:57:50 dist-test-slave-tz51 krb5kdc[4768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506670, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:50.558241 4781 init.cc:377] Logged in from keytab as kudu/127.14.68.62@KRBTEST.COM (short username kudu)
I20260927 10:57:50.559135 4781 webserver.cc:460] Webserver started at http://127.14.68.62:41835/ using document root <none> and password file <none>
I20260927 10:57:50.559738 4781 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:50.559835 4781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:50.560065 4781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:50.561998 4781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-0/data/instance:
uuid: "d2919480d95a46b29b0766d1761c2471"
format_stamp: "Formatted at 2026-09-27 10:57:50 on dist-test-slave-tz51"
server_key: "377407664a41b1f901c514faea151532"
server_key_iv: "0853d469f16e025614f6b408b2e2aecc"
server_key_version: "encryptionkey@0"
I20260927 10:57:50.562546 4781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-0/wal/instance:
uuid: "d2919480d95a46b29b0766d1761c2471"
format_stamp: "Formatted at 2026-09-27 10:57:50 on dist-test-slave-tz51"
server_key: "377407664a41b1f901c514faea151532"
server_key_iv: "0853d469f16e025614f6b408b2e2aecc"
server_key_version: "encryptionkey@0"
I20260927 10:57:50.566192 4781 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:57:50.568504 4796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:50.569645 4781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:57:50.569790 4781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-0/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-0/wal
uuid: "d2919480d95a46b29b0766d1761c2471"
format_stamp: "Formatted at 2026-09-27 10:57:50 on dist-test-slave-tz51"
server_key: "377407664a41b1f901c514faea151532"
server_key_iv: "0853d469f16e025614f6b408b2e2aecc"
server_key_version: "encryptionkey@0"
I20260927 10:57:50.569895 4781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:50.593057 4781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:50.595978 4781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:50.596175 4781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:50.603111 4781 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.62:41243
I20260927 10:57:50.603452 4848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.62:41243 every 8 connection(s)
I20260927 10:57:50.604112 4781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:50.606778 4849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:50.609952 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 4781
I20260927 10:57:50.610073 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-0/wal/instance
I20260927 10:57:50.610380 14608 external_mini_cluster.cc:1428] Setting key 1d5e2d4c606b9bd32bef3ed0c03f3f18
I20260927 10:57:50.610750 4849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41243 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.68.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.61@KRBTEST.COM" created.
Sep 27 10:57:50 dist-test-slave-tz51 krb5kdc[4768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506670, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:57:50.627449 4853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.614249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57584 (local address 127.14.68.62:41243)
0927 10:57:50.614606 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:50.614619 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:50.614667 (+ 48us) server_negotiation.cc:407] Connection header received
0927 10:57:50.615184 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:50.615193 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:50.615477 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:50.615722 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:50.616566 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:50.618035 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:50.619120 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:50.619517 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:50.619638 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:50.622332 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:50.622343 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:50.622352 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:50.622372 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:50.624377 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:50.624944 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:50.624949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:50.624950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:50.625057 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:50.625436 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:50.625438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:50.625439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:50.625811 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:50.625916 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:50.626407 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 10:57:50.626669 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":248,"thread_start_us":97,"threads_started":1}
I20260927 10:57:50.627452 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.613383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:41243 (local address 127.0.0.1:57584)
0927 10:57:50.613841 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:50.613895 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:50.614841 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:50.615752 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:50.615764 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:50.616147 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:50.616403 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:50.616416 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:50.618177 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:50.618180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:50.618978 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:50.618987 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:50.619659 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:50.619662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:50.619832 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:50.620436 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:50.620452 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:50.622185 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:50.624485 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:50.624497 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:50.624510 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:50.624831 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:50.625174 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:50.625180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:50.625185 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:50.625338 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:50.625933 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:50.625941 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:50.626130 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 10:57:50.626341 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 10:57:50.626751 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":78,"thread_start_us":132,"threads_started":1}
I20260927 10:57:50.629849 4849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:57:50.630559 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.630299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:42369 (local address 127.0.0.1:34800)
0927 10:57:50.630439 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:50.630508 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.61:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 10:57:50.630795 4797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.68.61:42369: connect: Connection refused (error 111)
I20260927 10:57:50.631515 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.631342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:42369 (local address 127.0.0.1:34812)
0927 10:57:50.631431 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:50.631471 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.61:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 10:57:50.631739 4849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.61:42369: Network error: Client connection negotiation failed: client connection to 127.14.68.61:42369: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.61.keytab.
Entry for principal kudu/127.14.68.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.61.keytab.
I20260927 10:57:50.636551 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.61:42369
--webserver_interface=127.14.68.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.61.keytab
--principal=kudu/127.14.68.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:43721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:41243,127.14.68.61:42369,127.14.68.60:33573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:57:50.681022 4849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } attempt: 1
I20260927 10:57:50.681911 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.681615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:42369 (local address 127.0.0.1:34818)
0927 10:57:50.681771 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:50.681836 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.61:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 10:57:50.682614 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.682409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:42369 (local address 127.0.0.1:34822)
0927 10:57:50.682514 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:50.682564 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.61:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 10:57:50.682766 4849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.61:42369: Network error: Client connection negotiation failed: client connection to 127.14.68.61:42369: connect: Connection refused (error 111)
W20260927 10:57:50.745776 4856 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:57:50.746081 4856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:50.746107 4856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:50.746212 4856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:50.750495 4856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:50.750546 4856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:50.750567 4856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:50.750625 4856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:50.750650 4856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:50.750943 4849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } attempt: 2
I20260927 10:57:50.751814 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.751426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:42369 (local address 127.0.0.1:34830)
0927 10:57:50.751614 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:50.751680 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.61:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 10:57:50.752548 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.752354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:42369 (local address 127.0.0.1:34832)
0927 10:57:50.752452 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:50.752498 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.61:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 10:57:50.752686 4849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.61:42369: Network error: Client connection negotiation failed: client connection to 127.14.68.61:42369: connect: Connection refused (error 111)
I20260927 10:57:50.756275 4856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:43721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:41243,127.14.68.61:42369,127.14.68.60:33573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.61.keytab
--principal=kudu/127.14.68.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.61:42369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.68.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:50.757192 4856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:50.758095 4856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:50.766363 4864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:50.766376 4861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:50.766381 4862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:50.766433 4856 server_base.cc:1034] running on GCE node
I20260927 10:57:50.767119 4856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:50.768213 4856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:50.769402 4856 hybrid_clock.cc:648] HybridClock initialized: now 1790506670769388 us; error 24 us; skew 500 ppm
Sep 27 10:57:50 dist-test-slave-tz51 krb5kdc[4768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506670, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:50.772101 4856 init.cc:377] Logged in from keytab as kudu/127.14.68.61@KRBTEST.COM (short username kudu)
I20260927 10:57:50.772904 4856 webserver.cc:460] Webserver started at http://127.14.68.61:33431/ using document root <none> and password file <none>
I20260927 10:57:50.773507 4856 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:50.773593 4856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:50.773864 4856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:50.776081 4856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-1/data/instance:
uuid: "a0ffc6e2bec8466294244929c70f1c67"
format_stamp: "Formatted at 2026-09-27 10:57:50 on dist-test-slave-tz51"
server_key: "c12033aeb67d6a2b1e7d7e3388188d20"
server_key_iv: "5cd9fae7c3d45952d28826930d873498"
server_key_version: "encryptionkey@0"
I20260927 10:57:50.776626 4856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-1/wal/instance:
uuid: "a0ffc6e2bec8466294244929c70f1c67"
format_stamp: "Formatted at 2026-09-27 10:57:50 on dist-test-slave-tz51"
server_key: "c12033aeb67d6a2b1e7d7e3388188d20"
server_key_iv: "5cd9fae7c3d45952d28826930d873498"
server_key_version: "encryptionkey@0"
I20260927 10:57:50.780390 4856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 10:57:50.782802 4871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:50.784072 4856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:57:50.784201 4856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-1/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-1/wal
uuid: "a0ffc6e2bec8466294244929c70f1c67"
format_stamp: "Formatted at 2026-09-27 10:57:50 on dist-test-slave-tz51"
server_key: "c12033aeb67d6a2b1e7d7e3388188d20"
server_key_iv: "5cd9fae7c3d45952d28826930d873498"
server_key_version: "encryptionkey@0"
I20260927 10:57:50.784300 4856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:50.797276 4856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:50.799907 4856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:50.800102 4856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:50.807292 4856 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.61:42369
I20260927 10:57:50.807574 4923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.61:42369 every 8 connection(s)
I20260927 10:57:50.808352 4856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-1/data/info.pb
I20260927 10:57:50.811781 4924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:50.812463 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 4856
I20260927 10:57:50.812584 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-1/wal/instance
I20260927 10:57:50.812947 14608 external_mini_cluster.cc:1428] Setting key eb0a19849c57400134575419a232a70a
I20260927 10:57:50.817087 4924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41243 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.68.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.68.60@KRBTEST.COM" created.
Sep 27 10:57:50 dist-test-slave-tz51 krb5kdc[4768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506670, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.61@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:57:50.833737 4853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.819535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57592 (local address 127.14.68.62:41243)
0927 10:57:50.819756 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:50.819763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:50.820726 (+ 963us) server_negotiation.cc:407] Connection header received
0927 10:57:50.822004 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:50.822006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:50.822051 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:50.822128 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:50.823646 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:50.824359 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:50.826990 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:50.827232 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:50.827318 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:50.830152 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:50.830161 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:50.830163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:50.830186 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:50.831641 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:50.832189 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:50.832192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:50.832194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:50.832272 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:50.832605 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:50.832608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:50.832610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:50.832793 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:50.832896 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:50.833442 (+ 546us) server_negotiation.cc:299] Negotiation successful
0927 10:57:50.833568 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 10:57:50.834543 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.819972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:41243 (local address 127.0.0.1:57592)
0927 10:57:50.820507 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:50.820587 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:50.821778 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:50.822322 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:50.822334 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:50.822823 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:50.823480 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:50.823495 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:50.824481 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:50.824485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:50.825730 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:50.825744 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:50.827403 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:50.827409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:50.827624 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:50.828275 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:50.828300 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:50.829958 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:50.831799 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:50.831804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:50.831818 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:50.832065 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:50.832362 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:50.832366 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:50.832368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:50.832504 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:50.832894 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:50.832901 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:50.833141 (+ 240us) client_negotiation.cc:769] Sending connection context
0927 10:57:50.833381 (+ 240us) client_negotiation.cc:240] Negotiation successful
0927 10:57:50.833720 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":111,"threads_started":1}
I20260927 10:57:50.835915 4924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:57:50 dist-test-slave-tz51 krb5kdc[4768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506670, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.61@KRBTEST.COM for kudu/127.14.68.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.68.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.60.keytab.
Entry for principal kudu/127.14.68.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.60.keytab.
I20260927 10:57:50.842460 14608 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
/tmp/dist-test-taskAZZUsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.60:33573
--webserver_interface=127.14.68.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.60.keytab
--principal=kudu/127.14.68.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.68.20:43721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.68.62:41243,127.14.68.61:42369,127.14.68.60:33573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:57:50.843834 4849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } attempt: 3
I20260927 10:57:50.845010 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.836374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:42369 (local address 127.0.0.1:34834)
0927 10:57:50.836543 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:50.836556 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:50.836615 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:50.837113 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:50.837115 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:50.837239 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:50.837319 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:50.837324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:50.838251 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:50.838254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:50.838843 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:50.838852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:50.839423 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:50.839426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:50.839497 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:50.840093 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:50.840110 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:50.841235 (+ 1125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:50.843199 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:50.843205 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:50.843210 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:50.843477 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:50.843852 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:50.843858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:50.843859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:50.843955 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:50.844519 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:50.844521 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:50.844618 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 10:57:50.844724 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 10:57:50.844843 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 10:57:50.845150 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.836469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34834 (local address 127.14.68.61:42369)
0927 10:57:50.836748 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:50.836766 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:50.836783 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:57:50.836817 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:50.836821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:50.836957 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:50.837101 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:50.837414 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:50.838141 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:50.838943 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:50.839327 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:50.839408 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:50.841359 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:50.841386 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:50.841401 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:50.841427 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:50.843090 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:50.843604 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:50.843611 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:50.843614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:50.843734 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:50.844061 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:50.844064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:50.844065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:50.844351 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:50.844480 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:50.844820 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 10:57:50.844937 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":112,"threads_started":1}
I20260927 10:57:50.846946 4924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:57:50.847885 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.847434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:33573 (local address 127.0.0.1:51732)
0927 10:57:50.847676 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:50.847797 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260927 10:57:50.848517 4872 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.68.60:33573: connect: Connection refused (error 111)
I20260927 10:57:50.849308 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.849098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:33573 (local address 127.0.0.1:51734)
0927 10:57:50.849201 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:50.849252 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 10:57:50.849808 4924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.60:33573: Network error: Client connection negotiation failed: client connection to 127.14.68.60:33573: connect: Connection refused (error 111)
Sep 27 10:57:50 dist-test-slave-tz51 krb5kdc[4768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506670, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for kudu/127.14.68.61@KRBTEST.COM
I20260927 10:57:50.854930 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.844264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:42369 (local address 127.0.0.1:34846)
0927 10:57:50.844438 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:50.844453 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:50.844529 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:50.845961 (+ 1432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:50.845964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:50.846113 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:50.846214 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:50.846219 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:50.847191 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:50.847194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:50.848607 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:50.848618 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:50.849370 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:50.849373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:50.849485 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:50.849919 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:50.849933 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:50.851400 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:50.853180 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:50.853187 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:50.853191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:50.853470 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:50.853757 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:50.853764 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:50.853768 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:50.853876 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:50.854386 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:50.854389 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:50.854508 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 10:57:50.854621 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 10:57:50.854760 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 10:57:50.855490 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.844321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34846 (local address 127.14.68.61:42369)
0927 10:57:50.844742 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:50.844749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:50.844762 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:57:50.844798 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:50.844802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:50.844844 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:50.845906 (+ 1062us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:50.846348 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:50.847008 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:50.848737 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:50.849185 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:50.849357 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:50.851539 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:50.851542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:50.851544 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:50.851563 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:50.853064 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:50.853586 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:50.853589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:50.853590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:50.853664 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:50.853974 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:50.853976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:50.853977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:50.854193 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:50.854355 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:50.854742 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 10:57:50.854940 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":149,"threads_started":1}
I20260927 10:57:50.856942 4849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:57:50.858078 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.857797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:33573 (local address 127.0.0.1:51740)
0927 10:57:50.857958 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:50.858007 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 10:57:50.858855 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.858486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:33573 (local address 127.0.0.1:51752)
0927 10:57:50.858724 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:50.858780 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260927 10:57:50.859097 4849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.60:33573: Network error: Client connection negotiation failed: client connection to 127.14.68.60:33573: connect: Connection refused (error 111)
I20260927 10:57:50.890261 4849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } attempt: 1
I20260927 10:57:50.891108 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.890810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:33573 (local address 127.0.0.1:51766)
0927 10:57:50.890957 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:50.891026 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 10:57:50.891835 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.891584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:33573 (local address 127.0.0.1:51770)
0927 10:57:50.891690 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:50.891786 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 10:57:50.891966 4849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.60:33573: Network error: Client connection negotiation failed: client connection to 127.14.68.60:33573: connect: Connection refused (error 111)
I20260927 10:57:50.899580 4924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } attempt: 1
I20260927 10:57:50.900425 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.900135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:33573 (local address 127.0.0.1:51784)
0927 10:57:50.900269 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:50.900335 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 10:57:50.901237 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.900977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:33573 (local address 127.0.0.1:51786)
0927 10:57:50.901129 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:50.901181 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 10:57:50.901440 4924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.60:33573: Network error: Client connection negotiation failed: client connection to 127.14.68.60:33573: connect: Connection refused (error 111)
I20260927 10:57:50.967128 4849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } attempt: 2
I20260927 10:57:50.967959 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.967640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:33573 (local address 127.0.0.1:51794)
0927 10:57:50.967822 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:50.967888 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 10:57:50.968703 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.968489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:33573 (local address 127.0.0.1:51808)
0927 10:57:50.968595 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:50.968641 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 10:57:50.968863 4849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.60:33573: Network error: Client connection negotiation failed: client connection to 127.14.68.60:33573: connect: Connection refused (error 111)
I20260927 10:57:50.969594 4924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } attempt: 2
I20260927 10:57:50.970233 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.969962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:33573 (local address 127.0.0.1:51820)
0927 10:57:50.970115 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:50.970171 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 10:57:50.970296 4930 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:57:50.970626 4930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:50.970679 4930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:50.970748 4930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 10:57:50.970993 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:50.970805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:33573 (local address 127.0.0.1:51834)
0927 10:57:50.970901 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:50.970945 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.68.60:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 10:57:50.971199 4924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.68.60:33573: Network error: Client connection negotiation failed: client connection to 127.14.68.60:33573: connect: Connection refused (error 111)
W20260927 10:57:50.975087 4930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:50.975142 4930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:50.975167 4930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:50.975184 4930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:50.975236 4930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:50.980684 4930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.68.20:43721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.68.62:41243,127.14.68.61:42369,127.14.68.60:33573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/krb5kdc/kudu_127.14.68.60.keytab
--principal=kudu/127.14.68.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.68.60:33573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.68.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:57:50.981765 4930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:50.982805 4930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:50.991245 4937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:50.991245 4940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:50.991304 4938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:50.991335 4930 server_base.cc:1034] running on GCE node
I20260927 10:57:50.991991 4930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:50.993063 4930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:50.994251 4930 hybrid_clock.cc:648] HybridClock initialized: now 1790506670994237 us; error 24 us; skew 500 ppm
Sep 27 10:57:50 dist-test-slave-tz51 krb5kdc[4768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506670, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:50.996979 4930 init.cc:377] Logged in from keytab as kudu/127.14.68.60@KRBTEST.COM (short username kudu)
I20260927 10:57:50.997834 4930 webserver.cc:460] Webserver started at http://127.14.68.60:33621/ using document root <none> and password file <none>
I20260927 10:57:50.998363 4930 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:50.998421 4930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:50.998623 4930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:51.000653 4930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-2/data/instance:
uuid: "90ea53b9ab2645d3bb6bfed6895b7ab2"
format_stamp: "Formatted at 2026-09-27 10:57:50 on dist-test-slave-tz51"
server_key: "04f79a1bab2f2524694bd2655e57d9dc"
server_key_iv: "bcf9ebfb78fc4340a84b6d0f441f58c8"
server_key_version: "encryptionkey@0"
I20260927 10:57:51.001214 4930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-2/wal/instance:
uuid: "90ea53b9ab2645d3bb6bfed6895b7ab2"
format_stamp: "Formatted at 2026-09-27 10:57:50 on dist-test-slave-tz51"
server_key: "04f79a1bab2f2524694bd2655e57d9dc"
server_key_iv: "bcf9ebfb78fc4340a84b6d0f441f58c8"
server_key_version: "encryptionkey@0"
I20260927 10:57:51.004712 4930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:57:51.007014 4947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:51.008198 4930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:57:51.008316 4930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-2/data,/tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-2/wal
uuid: "90ea53b9ab2645d3bb6bfed6895b7ab2"
format_stamp: "Formatted at 2026-09-27 10:57:50 on dist-test-slave-tz51"
server_key: "04f79a1bab2f2524694bd2655e57d9dc"
server_key_iv: "bcf9ebfb78fc4340a84b6d0f441f58c8"
server_key_version: "encryptionkey@0"
I20260927 10:57:51.008415 4930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:51.017274 4930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:51.020586 4930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:51.020818 4930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:51.028185 4930 rpc_server.cc:307] RPC server started. Bound to: 127.14.68.60:33573
I20260927 10:57:51.028573 4999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.68.60:33573 every 8 connection(s)
I20260927 10:57:51.029197 4930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-2/data/info.pb
I20260927 10:57:51.029958 14608 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu as pid 4930
I20260927 10:57:51.030085 14608 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAZZUsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506389460539-14608-0/minicluster-data/master-2/wal/instance
I20260927 10:57:51.030431 14608 external_mini_cluster.cc:1428] Setting key 2eddb03181050f0e4361f84f747df3f6
I20260927 10:57:51.032562 5000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:51.036772 5000 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41243 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:57:51 dist-test-slave-tz51 krb5kdc[4768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.60@KRBTEST.COM
Sep 27 10:57:51 dist-test-slave-tz51 krb5kdc[4768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506670, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.60@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
I20260927 10:57:51.044946 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:51.031793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51836 (local address 127.14.68.60:33573)
0927 10:57:51.032183 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.032199 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.032234 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:57:51.032960 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.032980 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.033332 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.033612 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.034516 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.035490 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.036655 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.037150 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.037326 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:51.040184 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:51.040189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:51.040197 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:51.040223 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:51.041870 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:51.042313 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:51.042321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:51.042330 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:51.042473 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:51.042792 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:51.042797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:51.042800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:51.043227 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:51.043363 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:51.043764 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 10:57:51.044042 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":126,"threads_started":1}
I20260927 10:57:51.048206 4853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:51.038540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57608 (local address 127.14.68.62:41243)
0927 10:57:51.038689 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.038694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.039283 (+ 589us) server_negotiation.cc:407] Connection header received
0927 10:57:51.039419 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.039421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.039464 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.039529 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.040378 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.041161 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.042052 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.042289 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.042399 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:51.044725 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:51.044728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:51.044730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:51.044756 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:51.046317 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:51.046791 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:51.046795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:51.046796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:51.046886 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:51.047190 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:51.047196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:51.047201 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:51.047437 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:51.047544 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:51.047940 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 10:57:51.048048 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:57:51.048269 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:51.038410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.62:41243 (local address 127.0.0.1:57608)
0927 10:57:51.039141 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:51.039190 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:51.039325 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:51.039625 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:51.039632 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:51.040110 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:51.040256 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:51.040261 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.041287 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.041289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:51.041865 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:51.041873 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.042401 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.042404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:51.042489 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:51.043046 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:51.043057 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:51.044593 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:51.046421 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:51.046427 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:51.046431 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:51.046668 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:51.046970 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:51.046973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:51.046975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:51.047074 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:51.047554 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:51.047561 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:51.047671 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 10:57:51.047881 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 10:57:51.048049 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":150,"threads_started":1}
I20260927 10:57:51.049175 5000 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:57:51 dist-test-slave-tz51 krb5kdc[4768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.62@KRBTEST.COM
Sep 27 10:57:51 dist-test-slave-tz51 krb5kdc[4768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506670, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.60@KRBTEST.COM for kudu/127.14.68.61@KRBTEST.COM
I20260927 10:57:51.056095 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:51.047395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57610 (local address 127.14.68.62:41243)
0927 10:57:51.047771 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.047780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.047794 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:57:51.047825 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.047827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.047860 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.047952 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.048667 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.049187 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.050123 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.050370 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.050476 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:51.052874 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:51.052877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:51.052880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:51.052908 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:51.054448 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:51.054870 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:51.054873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:51.054875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:51.054961 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:51.055237 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:51.055241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:51.055242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:51.055414 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:51.055615 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:51.055781 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 10:57:51.055903 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":100,"threads_started":1}
I20260927 10:57:51.057842 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:51.049589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.61:42369 (local address 127.0.0.1:34862)
0927 10:57:51.049691 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:51.049705 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:51.049770 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:51.050057 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:51.050060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:51.050228 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:51.050307 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:51.050312 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.051219 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.051226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:51.051815 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:51.051823 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.052291 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.052294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:51.052391 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:51.052846 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:51.052859 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:51.054257 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:51.056146 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:51.056152 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:51.056156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:51.056427 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:51.056729 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:51.056732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:51.056734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:51.056822 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:51.057315 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:51.057318 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:51.057418 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 10:57:51.057520 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 10:57:51.057666 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 10:57:51.057868 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:51.049725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34862 (local address 127.14.68.61:42369)
0927 10:57:51.049863 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.049866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.049878 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:57:51.049912 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.049914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.049960 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.050046 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.050424 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.051074 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.051935 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.052180 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.052291 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:51.054399 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:51.054402 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:51.054404 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:51.054427 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:51.055911 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:51.056537 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:51.056541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:51.056543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:51.056623 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:51.056902 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:51.056905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:51.056906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:51.057159 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:51.057267 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:51.057588 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 10:57:51.057718 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 10:57:51.058722 5000 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:57:51.062355 4924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } attempt: 3
Sep 27 10:57:51 dist-test-slave-tz51 krb5kdc[4768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.68.61@KRBTEST.COM
Sep 27 10:57:51 dist-test-slave-tz51 krb5kdc[4768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506670, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.60@KRBTEST.COM for kudu/127.14.68.60@KRBTEST.COM
I20260927 10:57:51.067385 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:51.057290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34864 (local address 127.14.68.61:42369)
0927 10:57:51.057405 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.057409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.057429 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:57:51.057547 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.057550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.057595 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.057687 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.058339 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.059007 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.060146 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.060387 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.060497 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:51.063374 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:51.063378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:51.063381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:51.063406 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:51.065489 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:51.065875 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:51.065880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:51.065885 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:51.065971 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:51.066318 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:51.066321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:51.066323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:51.066529 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:51.066749 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:51.067103 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 10:57:51.067253 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 10:57:51 dist-test-slave-tz51 krb5kdc[4768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506670, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.61@KRBTEST.COM for kudu/127.14.68.60@KRBTEST.COM
I20260927 10:57:51.067999 4849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } attempt: 3
I20260927 10:57:51.068783 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:51.059179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:33573 (local address 127.0.0.1:51842)
0927 10:57:51.059300 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:51.059314 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:51.059398 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:51.059610 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:51.059613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:51.059795 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:51.059889 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:51.059895 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.061011 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.061028 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:51.061939 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:51.061950 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.062543 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.062549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:51.062659 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:51.063088 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:51.063101 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:51.064494 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:51.067216 (+ 2722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:51.067219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:51.067223 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:51.067471 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:51.067782 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:51.067786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:51.067788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:51.067882 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:51.068325 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:51.068328 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:51.068402 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 10:57:51.068502 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 10:57:51.068613 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":26}
I20260927 10:57:51.069911 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:51.059227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51842 (local address 127.14.68.60:33573)
0927 10:57:51.059320 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.059324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.059361 (+ 37us) server_negotiation.cc:407] Connection header received
0927 10:57:51.059476 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.059478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.059511 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.059571 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.059993 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.060896 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.062064 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.062411 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.062536 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:51.064600 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:51.064604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:51.064607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:51.064628 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:51.067110 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:51.067582 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:51.067586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:51.067590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:51.067673 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:51.067980 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:51.067983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:51.067985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:51.068195 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:51.068288 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:51.068961 (+ 673us) server_negotiation.cc:299] Negotiation successful
0927 10:57:51.069744 (+ 783us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":35}
I20260927 10:57:51.071601 14608 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:57:51.072700 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:51.063028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:33573 (local address 127.0.0.1:51852)
0927 10:57:51.063200 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:51.063216 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:51.063293 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:51.063810 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:51.063813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:51.063987 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:51.064079 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:51.064086 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.065058 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.065062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:51.065805 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:51.065813 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.066493 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.066496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:51.066561 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:51.066973 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:51.066985 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:51.068369 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:51.070832 (+ 2463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:51.070835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:51.070837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:51.071077 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:51.071679 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:51.071682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:51.071684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:51.071790 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:51.072250 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:51.072274 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:51.072365 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 10:57:51.072441 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 10:57:51.072561 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 10:57:51.073318 5000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90ea53b9ab2645d3bb6bfed6895b7ab2: Bootstrap starting.
Sep 27 10:57:51 dist-test-slave-tz51 krb5kdc[4768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506670, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.68.62@KRBTEST.COM for kudu/127.14.68.60@KRBTEST.COM
I20260927 10:57:51.074257 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:51.063102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51852 (local address 127.14.68.60:33573)
0927 10:57:51.063560 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.063566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.063576 (+ 10us) server_negotiation.cc:407] Connection header received
0927 10:57:51.063602 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.063604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.063681 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.063793 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.064223 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.064968 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.065926 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.066158 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.066284 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:51.068764 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:51.068767 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:51.068769 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:51.068793 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:51.070716 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:51.071494 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:51.071497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:51.071498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:51.071576 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:51.071907 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:51.071913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:51.071918 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:51.072121 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:51.072935 (+ 814us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:51.073006 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 10:57:51.073110 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":167,"threads_started":1}
I20260927 10:57:51.073067 4823 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d2919480d95a46b29b0766d1761c2471"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57610
I20260927 10:57:51.076226 5000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90ea53b9ab2645d3bb6bfed6895b7ab2: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:51.077134 5000 log.cc:824] T 00000000000000000000000000000000 P 90ea53b9ab2645d3bb6bfed6895b7ab2: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:51.077407 4924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0ffc6e2bec8466294244929c70f1c67: Bootstrap starting.
I20260927 10:57:51.077773 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:51.068358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.68.60:33573 (local address 127.0.0.1:51854)
0927 10:57:51.068456 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:51.068471 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:51.068549 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:51.069419 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:51.069422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:51.069641 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:51.069771 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:51.069781 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.070921 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.070924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:51.071511 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:51.071521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.072166 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.072170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:51.072273 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:51.072689 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:51.072715 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:51.074047 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:51.076272 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:51.076277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:51.076280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:51.076541 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:51.076850 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:51.076853 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:51.076855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:51.076951 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:51.077336 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:51.077339 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:51.077439 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 10:57:51.077554 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 10:57:51.077667 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 10:57:51.078287 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:51.068598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51854 (local address 127.14.68.60:33573)
0927 10:57:51.069222 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.069231 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.069243 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:57:51.069270 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.069272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.069317 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.069398 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.069907 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.070798 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.071645 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.072026 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.072486 (+ 460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:51.074195 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:51.074199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:51.074201 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:51.074228 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:51.076151 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:51.076649 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:51.076653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:51.076655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:51.076748 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:51.077053 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:51.077056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:51.077058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:51.077247 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:51.077342 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:51.078084 (+ 742us) server_negotiation.cc:299] Negotiation successful
0927 10:57:51.078179 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":124,"threads_started":1}
I20260927 10:57:51.079243 5000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90ea53b9ab2645d3bb6bfed6895b7ab2: No bootstrap required, opened a new log
I20260927 10:57:51.079974 4924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0ffc6e2bec8466294244929c70f1c67: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:51.080549 4849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471: Bootstrap starting.
I20260927 10:57:51.081243 4924 log.cc:824] T 00000000000000000000000000000000 P a0ffc6e2bec8466294244929c70f1c67: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:51.082547 4849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:51.083165 4849 log.cc:824] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:51.084200 5000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90ea53b9ab2645d3bb6bfed6895b7ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2919480d95a46b29b0766d1761c2471" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41243 } } peers { permanent_uuid: "a0ffc6e2bec8466294244929c70f1c67" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } } peers { permanent_uuid: "90ea53b9ab2645d3bb6bfed6895b7ab2" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } }
I20260927 10:57:51.084441 5000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90ea53b9ab2645d3bb6bfed6895b7ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90ea53b9ab2645d3bb6bfed6895b7ab2, State: Initialized, Role: FOLLOWER
I20260927 10:57:51.084734 4924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0ffc6e2bec8466294244929c70f1c67: No bootstrap required, opened a new log
I20260927 10:57:51.084771 5000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90ea53b9ab2645d3bb6bfed6895b7ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2919480d95a46b29b0766d1761c2471" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41243 } } peers { permanent_uuid: "a0ffc6e2bec8466294244929c70f1c67" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } } peers { permanent_uuid: "90ea53b9ab2645d3bb6bfed6895b7ab2" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } }
I20260927 10:57:51.085170 4849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471: No bootstrap required, opened a new log
I20260927 10:57:51.085358 5011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90ea53b9ab2645d3bb6bfed6895b7ab2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2919480d95a46b29b0766d1761c2471" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41243 } } peers { permanent_uuid: "a0ffc6e2bec8466294244929c70f1c67" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } } peers { permanent_uuid: "90ea53b9ab2645d3bb6bfed6895b7ab2" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } } }
I20260927 10:57:51.085662 5011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90ea53b9ab2645d3bb6bfed6895b7ab2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:57:51.086047 5000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90ea53b9ab2645d3bb6bfed6895b7ab2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:51.088265 4849 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2919480d95a46b29b0766d1761c2471" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41243 } } peers { permanent_uuid: "a0ffc6e2bec8466294244929c70f1c67" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } } peers { permanent_uuid: "90ea53b9ab2645d3bb6bfed6895b7ab2" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } }
I20260927 10:57:51.088474 4849 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2919480d95a46b29b0766d1761c2471, State: Initialized, Role: FOLLOWER
I20260927 10:57:51.088781 4849 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2919480d95a46b29b0766d1761c2471" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41243 } } peers { permanent_uuid: "a0ffc6e2bec8466294244929c70f1c67" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } } peers { permanent_uuid: "90ea53b9ab2645d3bb6bfed6895b7ab2" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } }
I20260927 10:57:51.089285 5023 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2919480d95a46b29b0766d1761c2471" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41243 } } peers { permanent_uuid: "a0ffc6e2bec8466294244929c70f1c67" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } } peers { permanent_uuid: "90ea53b9ab2645d3bb6bfed6895b7ab2" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } } }
I20260927 10:57:51.089411 5023 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:57:51.090024 4849 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:51.090073 4823 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 10:57:51.090044 4924 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0ffc6e2bec8466294244929c70f1c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2919480d95a46b29b0766d1761c2471" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41243 } } peers { permanent_uuid: "a0ffc6e2bec8466294244929c70f1c67" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } } peers { permanent_uuid: "90ea53b9ab2645d3bb6bfed6895b7ab2" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } }
I20260927 10:57:51.090155 4823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:51.090245 4924 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0ffc6e2bec8466294244929c70f1c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0ffc6e2bec8466294244929c70f1c67, State: Initialized, Role: FOLLOWER
I20260927 10:57:51.090523 4924 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0ffc6e2bec8466294244929c70f1c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2919480d95a46b29b0766d1761c2471" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41243 } } peers { permanent_uuid: "a0ffc6e2bec8466294244929c70f1c67" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } } peers { permanent_uuid: "90ea53b9ab2645d3bb6bfed6895b7ab2" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } }
I20260927 10:57:51.091512 4823 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2919480d95a46b29b0766d1761c2471" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41243 } } peers { permanent_uuid: "a0ffc6e2bec8466294244929c70f1c67" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } } peers { permanent_uuid: "90ea53b9ab2645d3bb6bfed6895b7ab2" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } }
I20260927 10:57:51.091614 4924 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0ffc6e2bec8466294244929c70f1c67 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:51.089645 4930 master_runner.cc:186] Error getting master registration for 127.14.68.62:41243: OK, instance_id { permanent_uuid: "d2919480d95a46b29b0766d1761c2471" instance_seqno: 1790506670601208 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:57:51.093009 4930 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:57:51.092469 4823 leader_election.cc:288] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471 [CANDIDATE]: Term 1 election: Requested vote from peers a0ffc6e2bec8466294244929c70f1c67 (127.14.68.61:42369), 90ea53b9ab2645d3bb6bfed6895b7ab2 (127.14.68.60:33573)
I20260927 10:57:51.093637 4898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2919480d95a46b29b0766d1761c2471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a0ffc6e2bec8466294244929c70f1c67"
I20260927 10:57:51.093855 4898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0ffc6e2bec8466294244929c70f1c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:51.095124 4898 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a0ffc6e2bec8466294244929c70f1c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2919480d95a46b29b0766d1761c2471 in term 1.
I20260927 10:57:51.095539 4974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2919480d95a46b29b0766d1761c2471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "90ea53b9ab2645d3bb6bfed6895b7ab2"
I20260927 10:57:51.095772 4974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90ea53b9ab2645d3bb6bfed6895b7ab2 [term 0 FOLLOWER]: Advancing to term 1
W20260927 10:57:51.095649 5037 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:57:51.095885 5037 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:57:51.097107 4974 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 90ea53b9ab2645d3bb6bfed6895b7ab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2919480d95a46b29b0766d1761c2471 in term 1.
I20260927 10:57:51.097770 4797 leader_election.cc:302] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90ea53b9ab2645d3bb6bfed6895b7ab2, d2919480d95a46b29b0766d1761c2471; no voters:
W20260927 10:57:51.098124 4802 tablet.cc:2369] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:57:51.099501 5051 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0ffc6e2bec8466294244929c70f1c67: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:57:51.099572 5051 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:57:51.102370 4856 master_runner.cc:263] No leader master found from master a0ffc6e2bec8466294244929c70f1c67
I20260927 10:57:51.102435 4856 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:57:51.102648 5023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:51.102922 5023 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471 [term 1 LEADER]: Becoming Leader. State: Replica: d2919480d95a46b29b0766d1761c2471, State: Running, Role: LEADER
I20260927 10:57:51.103256 5023 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2919480d95a46b29b0766d1761c2471" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41243 } } peers { permanent_uuid: "a0ffc6e2bec8466294244929c70f1c67" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } } peers { permanent_uuid: "90ea53b9ab2645d3bb6bfed6895b7ab2" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } }
I20260927 10:57:51.104240 4781 master_runner.cc:263] No leader master found from master d2919480d95a46b29b0766d1761c2471
I20260927 10:57:51.104297 4781 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:57:51.105082 5023 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2919480d95a46b29b0766d1761c2471. Latest consensus state: current_term: 1 leader_uuid: "d2919480d95a46b29b0766d1761c2471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2919480d95a46b29b0766d1761c2471" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41243 } } peers { permanent_uuid: "a0ffc6e2bec8466294244929c70f1c67" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } } peers { permanent_uuid: "90ea53b9ab2645d3bb6bfed6895b7ab2" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } } }
I20260927 10:57:51.105191 5023 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471 [sys.catalog]: This master's current role is: LEADER
W20260927 10:57:51.106379 5022 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90ea53b9ab2645d3bb6bfed6895b7ab2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:57:51.106514 5022 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:57:51.108445 5024 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0ffc6e2bec8466294244929c70f1c67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2919480d95a46b29b0766d1761c2471" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41243 } } peers { permanent_uuid: "a0ffc6e2bec8466294244929c70f1c67" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } } peers { permanent_uuid: "90ea53b9ab2645d3bb6bfed6895b7ab2" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } } }
I20260927 10:57:51.108647 5024 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0ffc6e2bec8466294244929c70f1c67 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:57:51.112722 5056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:51.113816 5056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:51.122298 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:51.107726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34880 (local address 127.14.68.61:42369)
0927 10:57:51.107836 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.107840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.112344 (+ 4504us) server_negotiation.cc:407] Connection header received
0927 10:57:51.113420 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.113423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.113470 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.113521 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.114442 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.115210 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.116394 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.116728 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.116798 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:51.118851 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:51.118855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:51.118857 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:51.118881 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:51.120533 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:51.121001 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:51.121005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:51.121007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:51.121082 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:51.121387 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:51.121390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:51.121392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:51.121591 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:51.121698 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:51.122022 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 10:57:51.122133 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:57:51.124882 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:51.107104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57620 (local address 127.14.68.62:41243)
0927 10:57:51.107231 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.107237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.112016 (+ 4779us) server_negotiation.cc:407] Connection header received
0927 10:57:51.112152 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.112155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.112248 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.112327 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.113280 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.114085 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.118576 (+ 4491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.118961 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.119037 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:51.121230 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:51.121235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:51.121238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:51.121264 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:51.122974 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:51.123492 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:51.123496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:51.123497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:51.123579 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:51.123953 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:51.123956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:51.123958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:51.124195 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:51.124306 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:51.124624 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 10:57:51.124737 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:57:51.130623 4974 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90ea53b9ab2645d3bb6bfed6895b7ab2 [term 1 FOLLOWER]: Refusing update from remote peer d2919480d95a46b29b0766d1761c2471: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:51.131340 5053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90ea53b9ab2645d3bb6bfed6895b7ab2" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:51.132148 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:51.108037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51866 (local address 127.14.68.60:33573)
0927 10:57:51.108149 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.108154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.117416 (+ 9262us) server_negotiation.cc:407] Connection header received
0927 10:57:51.117555 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.117558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.117605 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.117680 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.124319 (+ 6639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.125105 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.126167 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:51.126496 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:51.126572 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:51.128282 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:51.128285 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:51.128288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:51.128314 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:51.129980 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:51.130773 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:51.130776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:51.130778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:51.130858 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:51.131162 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:51.131166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:51.131168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:51.131383 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:51.131511 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:51.131892 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 10:57:51.132012 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 10:57:51.135506 5011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90ea53b9ab2645d3bb6bfed6895b7ab2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2919480d95a46b29b0766d1761c2471. Latest consensus state: current_term: 1 leader_uuid: "d2919480d95a46b29b0766d1761c2471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2919480d95a46b29b0766d1761c2471" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41243 } } peers { permanent_uuid: "a0ffc6e2bec8466294244929c70f1c67" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } } peers { permanent_uuid: "90ea53b9ab2645d3bb6bfed6895b7ab2" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } } }
I20260927 10:57:51.135638 5011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90ea53b9ab2645d3bb6bfed6895b7ab2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:57:51.136901 4898 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a0ffc6e2bec8466294244929c70f1c67 [term 1 FOLLOWER]: Refusing update from remote peer d2919480d95a46b29b0766d1761c2471: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:51.137574 5053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0ffc6e2bec8466294244929c70f1c67" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:51.141758 5024 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0ffc6e2bec8466294244929c70f1c67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2919480d95a46b29b0766d1761c2471. Latest consensus state: current_term: 1 leader_uuid: "d2919480d95a46b29b0766d1761c2471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2919480d95a46b29b0766d1761c2471" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41243 } } peers { permanent_uuid: "a0ffc6e2bec8466294244929c70f1c67" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } } peers { permanent_uuid: "90ea53b9ab2645d3bb6bfed6895b7ab2" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } } }
I20260927 10:57:51.141891 5024 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0ffc6e2bec8466294244929c70f1c67 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:57:51.152505 5053 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d2919480d95a46b29b0766d1761c2471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2919480d95a46b29b0766d1761c2471" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41243 } } peers { permanent_uuid: "a0ffc6e2bec8466294244929c70f1c67" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } } peers { permanent_uuid: "90ea53b9ab2645d3bb6bfed6895b7ab2" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } } }
I20260927 10:57:51.152678 5053 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:51.152788 5060 mvcc.cc:205] Tried to move back new op lower bound from 7333915324941676544 to 7333915324842663936. Current Snapshot: MvccSnapshot[applied={T|T < 7333915324941676544}]
I20260927 10:57:51.154103 5024 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0ffc6e2bec8466294244929c70f1c67 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d2919480d95a46b29b0766d1761c2471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2919480d95a46b29b0766d1761c2471" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41243 } } peers { permanent_uuid: "a0ffc6e2bec8466294244929c70f1c67" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } } peers { permanent_uuid: "90ea53b9ab2645d3bb6bfed6895b7ab2" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } } }
I20260927 10:57:51.154222 5024 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0ffc6e2bec8466294244929c70f1c67 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:57:51.157648 5063 mvcc.cc:205] Tried to move back new op lower bound from 7333915324941676544 to 7333915324842663936. Current Snapshot: MvccSnapshot[applied={T|T < 7333915324941676544}]
I20260927 10:57:51.158391 5023 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d2919480d95a46b29b0766d1761c2471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2919480d95a46b29b0766d1761c2471" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41243 } } peers { permanent_uuid: "a0ffc6e2bec8466294244929c70f1c67" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } } peers { permanent_uuid: "90ea53b9ab2645d3bb6bfed6895b7ab2" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } } }
I20260927 10:57:51.158515 5023 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:51.159510 5011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90ea53b9ab2645d3bb6bfed6895b7ab2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d2919480d95a46b29b0766d1761c2471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2919480d95a46b29b0766d1761c2471" member_type: VOTER last_known_addr { host: "127.14.68.62" port: 41243 } } peers { permanent_uuid: "a0ffc6e2bec8466294244929c70f1c67" member_type: VOTER last_known_addr { host: "127.14.68.61" port: 42369 } } peers { permanent_uuid: "90ea53b9ab2645d3bb6bfed6895b7ab2" member_type: VOTER last_known_addr { host: "127.14.68.60" port: 33573 } } }
I20260927 10:57:51.159639 5011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90ea53b9ab2645d3bb6bfed6895b7ab2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:57:51.159984 5061 mvcc.cc:205] Tried to move back new op lower bound from 7333915324941676544 to 7333915324842663936. Current Snapshot: MvccSnapshot[applied={T|T < 7333915324941676544}]
I20260927 10:57:51.162045 5056 catalog_manager.cc:1347] Generated new cluster ID: 8bc489f6d6c241b9b79e8ae89a805c17
I20260927 10:57:51.162119 5056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:51.217747 5056 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:51.219128 5056 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:51.244977 5056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2919480d95a46b29b0766d1761c2471: Generated new TSK 0
I20260927 10:57:51.245770 5056 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:57:51.282514 5005 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:51.279215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57624 (local address 127.14.68.62:41243)
0927 10:57:51.279349 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.279354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.281101 (+ 1747us) server_negotiation.cc:407] Connection header received
0927 10:57:51.281565 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.281569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.281628 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.281701 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.282376 (+ 675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57624: BlockingRecv error: recv got EOF from 127.0.0.1:57624 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 10:57:51.284000 4929 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:51.280048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34882 (local address 127.14.68.61:42369)
0927 10:57:51.280175 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.280180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.283034 (+ 2854us) server_negotiation.cc:407] Connection header received
0927 10:57:51.283223 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.283226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.283279 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.283337 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.283882 (+ 545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34882: BlockingRecv error: recv got EOF from 127.0.0.1:34882 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 10:57:51.285415 5007 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:51.280731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51870 (local address 127.14.68.60:33573)
0927 10:57:51.280853 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.280857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.284379 (+ 3522us) server_negotiation.cc:407] Connection header received
0927 10:57:51.284649 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.284652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.284718 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.284775 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.285298 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51870: BlockingRecv error: recv got EOF from 127.0.0.1:51870 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 10:57:51.290272 4929 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:51.288626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34894 (local address 127.14.68.61:42369)
0927 10:57:51.288770 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.288774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.289292 (+ 518us) server_negotiation.cc:407] Connection header received
0927 10:57:51.289520 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.289524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.289597 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.289653 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.290172 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34894: BlockingRecv error: recv got EOF from 127.0.0.1:34894 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 10:57:51.295552 4929 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:51.293913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34910 (local address 127.14.68.61:42369)
0927 10:57:51.294019 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.294023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.294532 (+ 509us) server_negotiation.cc:407] Connection header received
0927 10:57:51.294763 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.294767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.294823 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.294887 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.295438 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34910: BlockingRecv error: recv got EOF from 127.0.0.1:34910 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 10:57:51.301543 4929 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:51.300030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34918 (local address 127.14.68.61:42369)
0927 10:57:51.300130 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.300134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.300586 (+ 452us) server_negotiation.cc:407] Connection header received
0927 10:57:51.300774 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.300777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.300826 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.300878 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.301420 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34918: BlockingRecv error: recv got EOF from 127.0.0.1:34918 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 10:57:51.308902 4929 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:51.307751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34934 (local address 127.14.68.61:42369)
0927 10:57:51.307879 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.307882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.308084 (+ 202us) server_negotiation.cc:407] Connection header received
0927 10:57:51.308196 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.308199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.308256 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.308346 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.308791 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34934: BlockingRecv error: recv got EOF from 127.0.0.1:34934 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 10:57:51.319952 4929 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:51.318778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34944 (local address 127.14.68.61:42369)
0927 10:57:51.318881 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.318886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.319100 (+ 214us) server_negotiation.cc:407] Connection header received
0927 10:57:51.319240 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.319243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.319294 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.319383 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.319848 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34944: BlockingRecv error: recv got EOF from 127.0.0.1:34944 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 10:57:51.339064 4929 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:51.338012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34956 (local address 127.14.68.61:42369)
0927 10:57:51.338140 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.338143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.338280 (+ 137us) server_negotiation.cc:407] Connection header received
0927 10:57:51.338375 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.338377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.338426 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.338496 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.338940 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34956: BlockingRecv error: recv got EOF from 127.0.0.1:34956 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 10:57:51.374388 4929 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:51.373102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34972 (local address 127.14.68.61:42369)
0927 10:57:51.373272 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.373276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.373441 (+ 165us) server_negotiation.cc:407] Connection header received
0927 10:57:51.373602 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.373605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.373657 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.373743 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.374262 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34972: BlockingRecv error: recv got EOF from 127.0.0.1:34972 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 10:57:51.441874 4929 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:51.440807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34986 (local address 127.14.68.61:42369)
0927 10:57:51.440932 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.440936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.441080 (+ 144us) server_negotiation.cc:407] Connection header received
0927 10:57:51.441189 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.441193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.441242 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.441352 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.441774 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34986: BlockingRecv error: recv got EOF from 127.0.0.1:34986 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 10:57:51.573206 4929 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:51.572024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34998 (local address 127.14.68.61:42369)
0927 10:57:51.572179 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.572184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.572326 (+ 142us) server_negotiation.cc:407] Connection header received
0927 10:57:51.572453 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.572456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.572504 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.572575 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.573065 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34998: BlockingRecv error: recv got EOF from 127.0.0.1:34998 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 10:57:51.832396 4929 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:51.831245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35012 (local address 127.14.68.61:42369)
0927 10:57:51.831409 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:51.831413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:51.831533 (+ 120us) server_negotiation.cc:407] Connection header received
0927 10:57:51.831671 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:51.831673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:51.831762 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:51.831831 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:51.832283 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35012: BlockingRecv error: recv got EOF from 127.0.0.1:35012 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 10:57:52.100883 5051 catalog_manager.cc:1259] Loaded cluster ID: 8bc489f6d6c241b9b79e8ae89a805c17
I20260927 10:57:52.100968 5051 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a0ffc6e2bec8466294244929c70f1c67: loading cluster ID for follower catalog manager: success
I20260927 10:57:52.102690 5051 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a0ffc6e2bec8466294244929c70f1c67: acquiring CA information for follower catalog manager: success
I20260927 10:57:52.104041 5051 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a0ffc6e2bec8466294244929c70f1c67: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:57:52.107676 5022 catalog_manager.cc:1259] Loaded cluster ID: 8bc489f6d6c241b9b79e8ae89a805c17
I20260927 10:57:52.107796 5022 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 90ea53b9ab2645d3bb6bfed6895b7ab2: loading cluster ID for follower catalog manager: success
I20260927 10:57:52.109432 5022 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 90ea53b9ab2645d3bb6bfed6895b7ab2: acquiring CA information for follower catalog manager: success
I20260927 10:57:52.110565 5022 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 90ea53b9ab2645d3bb6bfed6895b7ab2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:57:52.350605 5133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:52.346729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55654 (local address 127.14.68.61:42369)
0927 10:57:52.346993 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:52.346997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:52.347019 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:57:52.347161 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:52.347164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:52.347299 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:52.347409 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:57:52.347659 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.348223 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.349095 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.349345 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.349436 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:52.349706 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:57:52.350150 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:57:52.350329 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:52.350374 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 10:57:52.350432 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":103,"threads_started":1}
I20260927 10:57:52.357229 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:52.353656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54098 (local address 127.14.68.60:33573)
0927 10:57:52.353865 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:52.353869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:52.353982 (+ 113us) server_negotiation.cc:407] Connection header received
0927 10:57:52.354145 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:52.354147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:52.354264 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:52.354364 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:57:52.354538 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.355064 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.355908 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.356134 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.356226 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:52.356470 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:57:52.356895 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:57:52.356987 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:52.357040 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 10:57:52.357089 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":71,"threads_started":1}
I20260927 10:57:52.364349 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:52.360676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54100 (local address 127.14.68.60:33573)
0927 10:57:52.360801 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:52.360805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:52.360930 (+ 125us) server_negotiation.cc:407] Connection header received
0927 10:57:52.361093 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:52.361096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:52.361141 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:52.361219 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:57:52.361526 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.362019 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.363216 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.363537 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.363630 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:52.363905 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:57:52.363977 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:57:52.364049 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:52.364141 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 10:57:52.364198 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:57:52.364490 5133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:52.360307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55660 (local address 127.14.68.61:42369)
0927 10:57:52.360434 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:52.360438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:52.360730 (+ 292us) server_negotiation.cc:407] Connection header received
0927 10:57:52.360922 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:52.360925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:52.360978 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:52.361089 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:57:52.361429 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.362224 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.363334 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.363641 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.363781 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:52.363991 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:57:52.364079 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:57:52.364145 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:52.364285 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 10:57:52.364348 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:57:52.364775 5146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:52.360268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36630 (local address 127.14.68.62:41243)
0927 10:57:52.360505 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:52.360509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:52.360677 (+ 168us) server_negotiation.cc:407] Connection header received
0927 10:57:52.360784 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:52.360787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:52.360961 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:52.361075 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:57:52.361476 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.362254 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.363082 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.363408 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.363507 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:52.363862 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:57:52.364207 (+ 345us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:57:52.364290 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:52.364575 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 10:57:52.364632 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":62,"threads_started":1}
I20260927 10:57:52.368103 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 4781
I20260927 10:57:52.375064 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 4856
I20260927 10:57:52.381567 14608 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAZZUsq/build/debug/bin/kudu with pid 4930
2026-09-27T10:57:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3121 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6243 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (282916 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST